A multi-core neural network processor chip based on RISC-V architecture

Abstract

The present invention discloses a multi-core neural network processor chip of RISC-V architecture, including a main control module and a parallel computing cluster. The core of the main control module is a RISC-V 64GC instruction set processor, which is responsible for running an operating system, providing peripheral interface control and power management. The parallel computing cluster includes four RISC-V 32IMFC instruction set processors and a convolution acceleration engine. The RISC-V 32IMFC instruction set processor is responsible for running large-scale parallel vector calculations. The convolution acceleration engine is an operation unit matrix of a custom architecture, which provides acceleration for a convolutional neural network algorithm. The present invention has the characteristics of high computing power, low power consumption, and high parallelism, and can realize a neural network algorithm with relatively complex functions, large network scale, and relatively flexible connection within a reasonable chip area.

Description

Technical Field

[0001] The present invention relates to the field of processor chips, and in particular to a multi-core neural network processor chip with a RISC-V architecture, which can be used for low-power hardware implementation of neural network algorithms. Background Art

[0002] A deep neural network is a complex network system composed of a large number of simple components that are widely interconnected. It is characterized by distributed storage, asynchronous parallel processing, self-adaptation, self-learning, and fault tolerance, which makes it have great application prospects in sensory signal processing such as pattern recognition, speech analysis and synthesis, and computer vision. However, the implementation of neural network algorithms requires a lot of computing resources, especially large-scale algorithms, which are extremely inefficient when running on general-purpose CPUs.

[0003] In order to efficiently run neural network algorithms, people have to resort to GPUs or FPGAs, but in mobile scenarios or data edge scenarios, there is no sufficient power and device size support for GPUs. Using FPGAs requires the assistance of general-purpose CPUs, which has poor flexibility. Summary of the invention

[0004] The purpose of the present invention is to propose a multi-core neural network processor chip structure based on RISC-V architecture in view of the deficiencies of the prior art. As an open source reduced instruction set architecture, RISC-V architecture has the advantages of instruction set modularization and support for extended instructions, and is very suitable for the design of multi-core custom function processors.

[0005] The objective of the present invention is achieved through the following technical scheme: a multi-core neural network processor chip with a RISC-V architecture, the multi-core neural network processor chip is composed of a main control module and a parallel computing cluster, the two transmit data through cache mutual access, the parallel computing cluster has a hardware synchronization module, performs parallel thread scheduling and synchronization, and controls the clock gating of each core in the cluster; the main control module realizes control and feedback of the parallel computing cluster through the signal of the hardware synchronization module, and the main control module and the parallel computing cluster operate in different voltage domains and clock domains.

[0006] Furthermore, the main control module is mainly composed of a RISC-V64GC instruction set processor, an external interface, a power management module, an L1 cache, an L2 cache, a multi-channel I / O DMA, a clock module and a debugging module.

[0007] The RISC-V64GC instruction set processor instruction set in the main control module can control the data transmission of multiple external interfaces through multi-channel I / O DMA.

[0008] The power management module includes a power management unit PMU, a DC / DC converter and a low-dropout linear regulator LDO; the PMU controls the switching of the DC / DC and LDO through a signal line. After the PMU is turned on, it is initialized and outputs the power supply voltage of the main control module, which can then be controlled by the main control module.

[0009] The clock module is composed of an on-chip digitally controlled phase-locked loop, which generates an initial clock and provides it to the main control module, and then outputs an adjustable clock to the parallel computing cluster under the control of the main control module.

[0010] Furthermore, the L1 cache is a 32Kb SRAM, and the L2 cache is a 512kB SRAM.

[0011] Furthermore, the external interface includes a complete set of peripheral interfaces capable of capturing image, sound and vibration data in parallel, and a 4-channel PWM interface.

[0012] Furthermore, the data of peripheral input / output is managed by a multi-channel I / O DMA, thereby reducing the number of interactions and the workload of the control core when performing I / O.

[0013] Furthermore, the debugging module is composed of a debugging mode control signal line and a JTAG interface, and the program of the main control kernel is debugged through the module.

[0014] Furthermore, the parallel computing cluster consists of 4 RISC-V cores, a hardware synchronization module, a convolution acceleration engine, a multi-channel DMA, a shared instruction cache, a shared L2 cache and a debugging module, and works in an independent voltage domain and clock domain for executing computationally intensive program tasks.

[0015] The four RISC-V cores are of IMFC instruction set structure, run the same binary code, and use a shared instruction cache to store parallel programs.

[0016] The multi-channel DMA is used for batch data transmission from the shared L2 cache to the L2 cache of the main control module to reduce the access cost between modules.

[0017] The shared L2 cache is connected to four RISC-V cores via a set of crossbar switches.

[0018] The hardware synchronization module is responsible for fast event management, parallel thread scheduling and synchronization, and supports fine-grained parallelism. This module also controls the clock gating of each core in the cluster, and the core in the waiting state immediately enters the clock shielding state, making its dynamic power consumption zero.

[0019] In order to maximize energy efficiency and reduce the number of external components, the power management module deploys an on-chip DC / DC converter powered by an external power supply in the chip. When the chip is in working state, it can provide a voltage of 1.0V to 1.2V. When the chip is in sleep mode, the converter is turned off, leaving only a low-dropout linear regulator to power the clock, and selectively retaining part of the L2 memory power supply, specifically: only retaining the power supply of the bank with valid data in L2, closing other banks without data, to retain the real-time status of the application in order to wait for wake-up.

[0020] Furthermore, the debugging module is composed of a debugging mode control signal line and a JTAG interface, and the program code in the parallel computing cluster can be debugged through this module.

[0021] Furthermore, the RISC-V core adopts a sequential 4-stage pipeline structure, and each core has an L1 cache.

[0022] Furthermore, the convolution acceleration engine is composed of a processing unit PE array, a global cache and an on-chip network.

[0023] The global cache reads the data to be calculated from the L2 cache of the main control module through DMA control in the parallel computing cluster.

[0024] The on-chip network connects the global cache and all PEs, reads data from the global cache, and then packages it into data packets and sends it to the on-chip network. The on-chip network is a broadcast network, that is, all data will be sent to all PEs. The data packets carry ID numbers corresponding to the IDs of the PEs in the array. PEs only receive data packets with the same ID as themselves.

[0025] The PE array consists of 168 PEs, arranged in 12 rows and 14 columns. Each PE consists of a multiplier, SRAM and a control logic module. Each PE has a configurable ID number. After the PE receives a data packet from the on-chip network, it is stored in the SRAM. The control logic module controls the calculation process. After the data is read out, it is sent to the multiplier for calculation, and the result is returned to the SRAM. After completing a batch of calculations, all the result data in the PE is sent to the global cache through the on-chip network, and then sent to the L2 cache of the main control module through DMA.

[0026] Beneficial effects of the present invention:

[0027] 1. The main control module of the present invention can run the operating system, control the peripheral interface, control the computing cluster, and process general application programs.

[0028] 2. The convolution acceleration engine in the present invention can efficiently run convolution operations in neural networks.

[0029] 3. The parallel computing cluster in the present invention can efficiently execute a variety of computationally intensive programs, especially large-scale neural network algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the system architecture of the present invention;

[0031] Figure 2 This is a structural diagram of the convolution acceleration engine of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] like Figure 1 As shown, the chip of the present invention has two large module divisions, namely a main control module and a parallel computing cluster. The two transmit data through cache mutual access. The parallel computing cluster has a hardware synchronization module to perform parallel thread scheduling and synchronization and control the clock gating of each core in the cluster; the main control module realizes control and feedback of the parallel computing cluster through the signal of the hardware synchronization module. The main control module and the parallel computing cluster work in different voltage domains and clock domains, and adjust their respective voltages and clock frequencies according to the running load.

[0034] The main core of the main control module is RISC-V64GC architecture, that is, it supports I, M, A, F, D, C instruction subsets. The main control module is mainly composed of RISC-V64GC instruction set processor, external interface, power management module, L1 cache, L2 cache, multi-channel I / O DMA, clock module and debugging module. It has an L1 cache for storing intermediate data, a ROM for storing the main boot program, and a set of eFUSE storage keys to support secure boot mode. The L1 cache is a 32Kb SRAM, and the L2 cache is a 512kB SRAM.

[0035] The RISC-V64GC instruction set processor instruction set in the main control module can control the data transmission of multiple external interfaces through multi-channel I / O DMA. The clock module is composed of an on-chip digital control phase-locked loop, which generates an initial clock and provides it to the main control module, and then outputs an adjustable clock to the parallel computing cluster under the control of the main control module; the debugging module is composed of a debugging mode control signal line and a JTAG interface, and the program of the main control kernel is debugged through this module.

[0036] The main control module interface types include LVDS, UART, SPI, I2S, I2C, GPIO, PWM, which can capture image, sound and vibration data in parallel. The 4-channel PWM interface is used for motor control, valves and electric switches, etc. The main control core sends instructions to the multi-channel I / O DMA to control the data transmission of the interface input / output, thereby reducing the number of interactions between the core and the interface and the workload of the control core when executing I / O.

[0037] The power management module includes a power management unit (PMU), a DC / DC converter and a low dropout regulator (LDO). The PMU controls the switching of the DC / DC and LDO through signal lines. After the PMU is turned on, it is initialized and outputs the power supply voltage of the main control module, which can then be controlled by the main control module. The power module supplies power to all on-chip circuits, including the main control module and the parallel computing cluster.

[0038] The parallel computing cluster consists of 4 RISC-V cores, hardware synchronization module, convolution acceleration engine, multi-channel DMA, shared instruction cache, shared L2 cache and debugging module. It works in an independent voltage domain and clock domain and is used to execute computationally intensive program tasks.

[0039] The four RISC-V cores are of IMFC instruction set structure, run the same binary code, and use a shared instruction cache to store parallel programs.

[0040] The multi-channel DMA is used for batch data transmission from the shared L2 cache to the L2 cache of the main control module to reduce the access cost between modules.

[0041] The shared L2 cache is connected to four RISC-V cores via a set of crossbar switches.

[0042] The power management module provides 0.9 to 1.2V DC power to the main control core, and adjusts the power supply voltage according to different core loads. Provides three different drive voltages of 1.8V, 2.5V, and 3.3V to the interface. Provides 0.9 to 1.2V voltage to the parallel computing cluster. When the chip is in sleep mode, the converter is turned off, leaving only a low-dropout linear regulator LDO transformer to power the clock, and selectively retains part of the L2 memory power supply, specifically: only retain the power supply of the bank with valid data in L2, and close other banks without data to retain the real-time status of the application for fast wake-up. When the clock is active and there is no data to be retained, it can enter a deep sleep state. When the system processes more tasks or the computing tasks are heavy, the voltage is increased to run at a higher clock frequency.

[0043] The parallel computing cluster core is composed of four 32-bit RIS-V small cores, which adopt a sequential 4-stage pipeline structure. Each core has an L1 cache. Since it only runs computing tasks, the core only supports four sub-instruction sets: I, M, F, and C. It accelerates non-convolution operation-intensive applications, such as fast Fourier transform, discrete Fourier transform, and recursive filter algorithm. The main control core transmits the programs to be run by the four computing cores to the shared instruction cache through instructions, and then gives the four cores a reset command to let them read instructions from the shared instruction cache and start computing tasks. The data required for the calculation is obtained from the L2 cache of the main control module through DMA. The intermediate calculation results are stored in the shared L2 cache through crossbar, and the final calculation results are sent from the shared L2 cache to the L2 cache of the main control module by DMA.

[0044] The four RISC-V small cores in the parallel computing cluster are completely identical in structure, share an instruction cache, execute the same code, and are completely synchronized in behavior. They are connected to the L2 cache in the cluster through a crossbar. Due to the offset of the access address, they access different memory banks, so that they read four different sets of data during calculation.

[0045] In the calculation preparation stage, the main control module transfers the data to be calculated to the L2 in the cluster through the DMA of the computing cluster. The data is divided into 4 equal parts and stored in different bank locations. At the same time, the main control module also transfers the algorithm code to the shared instruction cache of the cluster through DMA.

[0046] During the calculation phase, the master core sends a synchronous reset signal to the four computing cores through the hardware synchronization unit, allowing them to simultaneously start reading the instructions in the instruction cache and start the calculation work. The four cores retrieve data from different banks to complete the calculation of the instruction code, and the intermediate result data is also stored in their corresponding banks. After completing all the calculation tasks, the four cores simultaneously send a completion signal, which is fed back to the master core through the hardware synchronization unit, and then enters the sleep state. The master core controls the reading back of the result data from the L2 cache in the cluster through DMA. A calculation task is completed. The structure of the parallel computing cluster has the following advantages: Such a shared instruction cache structure saves hardware usage and does not require an instruction space for each core. The small core structure only has the calculation function, which reduces the chip area. The kernel supports multiple computing instruction sets and has a high algorithm support coverage. The content of the shared instruction cache is burned by the master core, which can be compiled online and modified at will, and has high configurability. The fully synchronized calculation process does not require a memory read and write arbitration structure.

[0047] The hardware synchronization module is responsible for fast event management, scheduling of computing tasks, parallel thread scheduling and synchronization, and communication between the master control module and the parallel computing cluster. For example, the master control module sends a task start signal, and the cluster sends an interrupt signal to inform the master control that the task has been completed after completing the task. It supports fine-grained parallel operations and can operate the cluster core and convolution acceleration engine at the same time. This module also controls the clock gating of each computing module in the cluster. The core in the waiting state immediately enters the clock shielding state, making its dynamic power consumption zero.

[0048] The debugging module is composed of a debugging mode control signal line and a JTAG interface, and the program code in the parallel computing cluster can be debugged through this module.

[0049] The convolution acceleration engine consists of a processing element (PE) array, a global cache, and an on-chip network.

[0050] When the processor runs a deep neural network application, the computation of the convolution layer with large computational load can be transferred to a dedicated convolution acceleration engine, which is used to accelerate the convolution operation. It consists of an array of a large number of computing units, such as a 168 PE array with 12 rows and 14 columns.

[0051] First, the main control module sends a configuration instruction to configure the PE array into several blocks that match the size of the convolution kernel, such as several 3x3 or 5x5 blocks. Each PE is assigned an on-chip network ID according to the block scheme. PEs that receive the same data are assigned the same ID.

[0052] The on-chip network connects the global cache and each PE in the form of a broadcast network. According to the instructions, the data in the global cache is sent to the broadcast network in batches. Each time the data is sent, it will be received by all PEs, but PE only stores data that matches its own ID.

[0053] There are three SRAMs, a multiplier and a control logic circuit inside the PE. The three SRAMs store the inputs, weights and intermediate results of the neural network algorithm respectively. Each PE has a configurable ID number. After the PE receives the data packet from the on-chip network, it is stored in the SRAM. The control logic module controls the calculation process. After the data is read out, it is sent to the multiplier for calculation, and the result is returned to the SRAM. After completing a batch of calculations, all the result data in the PE is sent to the global cache through the on-chip network, and then sent to the L2 cache of the main control module through DMA. The controller reads the input, weights and intermediate results according to the pre-edited rules, and sends them to the multiplier for calculation until the final result is calculated and sent back to the interface of the on-chip network. Wait for the on-chip network to recycle the calculation results, and finally return the calculation results to the global cache.

[0054] The global cache reads the data to be calculated from the L2 cache of the main control module through DMA control in the parallel computing cluster, and the total capacity is 180Kb.

[0055] Since the PE array can perform hundreds of multiplication and addition operations per cycle, and array partitioning can reuse the calculation data, convolution operations can be completed more efficiently.

[0056] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A multi-core neural network processor chip based on RISC-V architecture, characterized in that: The multi-core neural network processor chip is composed of a main control module and a parallel computing cluster, which transmit data through cache mutual access. The main control module is composed of a RISC-V64GC instruction set processor, an external interface, a power management module, an L1 cache, an L2 cache, a multi-channel I / O DMA, a clock module and a debugging module; the parallel computing cluster is composed of four RISC-V cores, a hardware synchronization module, a convolution acceleration engine, a multi-channel DMA, a shared instruction cache, a shared L2 cache and a debugging module, and works in an independent voltage domain and clock domain for executing computationally intensive program tasks; the parallel computing cluster has a hardware synchronization module for parallel thread scheduling and synchronization, and controls the clock gating of each core in the cluster; The main control module realizes control and feedback of the parallel computing cluster through the signal of the hardware synchronization module. The main control module and the parallel computing cluster work in different voltage domains and clock domains.

2. A multi-core neural network processor chip based on RISC-V architecture according to claim 1, characterized in that: The RISC-V64GC instruction set processor instruction set in the main control module controls the data transmission of multiple external interfaces through multi-channel I / O DMA; The power management module includes a power management unit PMU, a DC / DC converter and a low-dropout linear regulator LDO; the PMU controls the switches of the DC / DC and the LDO through a signal line; the PMU is initialized after being turned on, outputs the power supply voltage of the main control module, and is then controlled by the main control module; The clock module is composed of an on-chip digitally controlled phase-locked loop, which generates an initial clock and provides it to the main control module, and then outputs an adjustable clock to the parallel computing cluster under the control of the main control module.

3. The multi-core neural network processor chip of RISC-V architecture according to claim 2, characterized in that: The L1 cache is a 32Kb SRAM and the L2 cache is a 512 kB SRAM.

4. A multi-core neural network processor chip based on RISC-V architecture according to claim 2, characterized in that: The external interface includes a complete set of peripheral interfaces capable of capturing image, sound and vibration data in parallel, as well as a 4-channel PWM interface.

5. The multi-core neural network processor chip of the RISC-V architecture according to claim 4, characterized in that: Data for peripheral input / output is managed by a multi-channel I / O DMA, thus reducing the number of interactions and the workload of the control core when performing I / O.

6. The multi-core neural network processor chip of the RISC-V architecture according to claim 2, characterized in that: The debugging module is composed of a debugging mode control signal line and a JTAG interface, and the program of the main control kernel is debugged through this module.

7. The multi-core neural network processor chip of the RISC-V architecture according to claim 1, characterized in that: The four RISC-V cores are of IMFC instruction set architecture, run the same binary code, and use a shared instruction cache to store parallel programs; The multi-channel DMA is used for batch data transmission from the shared L2 cache to the L2 cache of the main control module to reduce the access cost between modules; The shared L2 cache is connected to four RISC-V cores via a set of crossbar switches; The hardware synchronization module is responsible for fast event management, parallel thread scheduling and synchronization, and supports fine-grained parallelism; the module also controls the clock gating of each core in the cluster, and the core in the waiting state immediately enters the clock shielding state, making its dynamic power consumption zero; In order to maximize energy efficiency and reduce the number of external components, the power management module deploys an on-chip DC / DC converter powered by an external power supply in the chip; when the chip is in working state, it provides a voltage of 1.0V to 1.2V; when the chip is in sleep mode, the converter is turned off, leaving only a low-voltage difference linear regulator to power the clock, and selectively retaining part of the L2 memory power supply, specifically: only retaining the power supply of the bank with valid data in L2, closing other banks without data, to retain the real-time status of the application program in order to wait for wake-up.

8. The multi-core neural network processor chip of RISC-V architecture according to claim 7, characterized in that: The debugging module is composed of a debugging mode control signal line and a JTAG interface, and the program code in the parallel computing cluster is debugged through this module.

9. The multi-core neural network processor chip of RISC-V architecture according to claim 7, characterized in that: The RISC-V core adopts a sequential 4-stage pipeline structure, and each core has an L1 cache.

10. The multi-core neural network processor chip of RISC-V architecture according to claim 7, characterized in that: The convolution acceleration engine is composed of a processing unit PE array, a global cache and an on-chip network; The global cache reads the data to be calculated from the L2 cache of the main control module through DMA control in the parallel computing cluster; The on-chip network connects the global cache and all PEs, reads data from the global cache, and then packages it into data packets and sends it to the on-chip network. The on-chip network is a broadcast network, that is, all data will be sent to all PEs. The data packets carry ID numbers corresponding to the IDs of the PEs in the array; PEs only receive data packets with the same ID as their own. The PE array consists of 168 PEs arranged in 12 rows and 14 columns; each PE consists of a multiplier, an SRAM and a control logic module; each PE has a configurable ID number; After PE receives a data packet from the on-chip network, it is stored in the SRAM. The control logic module controls the calculation process. After the data is read out, it is sent to the multiplier and adder for calculation, and the result is returned to the SRAM. After completing a batch of calculations, all the result data in the PE is sent to the global cache through the on-chip network, and then sent to the L2 cache of the main control module through DMA.

Images