A convolutional neural network accelerator system based on a RISC-V processor

Abstract

A convolutional neural network accelerator system based on a RISC-V processor belongs to the technical field of neural network hardware acceleration. The RISC-V processor is a hummingbird E203 processor, and the convolutional neural network accelerator system is connected with the hummingbird E203 processor through a NICE interface. The NICE interface is used for realizing data interaction between the hummingbird E203 processor and the convolutional neural network accelerator system, and the data interaction includes transmission of custom extension instructions, memory data and result data. The convolutional neural network accelerator system comprises a weight configuration module, a sliding window generation module, a convolution calculation module, an activation module, a pooling module, a controller, a decoder, a data input module and a data output module. The application is used for realizing convolution calculation, activation processing and pooling processing in the convolutional neural network, and realizing collaborative control between the RISC-V processor and the accelerator.

Description

Technical Field

[0001] This invention relates to the field of neural network hardware acceleration technology, and in particular to a convolutional neural network accelerator system based on a RISC-V processor. Background Technology

[0002] Convolutional neural networks (CNNs) are an important class of deep learning models that have been widely applied in image recognition, object detection, edge sensing, and embedded smart terminals. A typical CNN usually includes operations such as convolution, activation, and pooling. Among these, convolution requires a large number of multiplication and addition calculations on the input feature map and the convolution kernel weights. This results in frequent data access and high computational density, which is a significant factor affecting the inference speed and energy consumption of neural networks.

[0003] In embedded applications, systems are typically limited by processor performance, storage bandwidth, chip area, and power consumption. Relying entirely on general-purpose processors to execute convolutional neural network inference tasks in software is not only computationally time-consuming but also incurs significant data transfer overhead, making it difficult to meet the demands of applications such as real-time image processing and low-power edge computing. Therefore, employing dedicated hardware architectures to accelerate key operations in convolutional neural networks has become an important technical means to improve the performance of embedded intelligent computing systems.

[0004] Existing hardware acceleration solutions for convolutional neural networks typically focus on optimizing convolution multiplication-accumulation operations, such as through parallel multiply-accumulate arrays, pipelined structures, or on-chip cache structures to improve convolution computation speed. However, in actual inference, the convolution results usually need to undergo activation and pooling processing before forming the data required by subsequent network layers. If convolution, activation, and pooling are performed in stages by different modules or software processes, intermediate feature data needs to be repeatedly moved between memory and computing units, which can easily increase the number of memory accesses and system latency, reducing the overall acceleration effect. In addition, some existing accelerators have relatively fixed configuration methods for convolution kernel weights and input feature maps, and the updating of convolution kernel parameters is not flexible enough, which is not conducive to adapting to the computational requirements of different network layers or different convolution tasks.

[0005] Therefore, it is necessary to propose a convolutional neural network acceleration scheme for RISC-V embedded processors, enabling accelerator parameter configuration, weight loading, feature map input, continuous convolution, activation, and pooling processing, and result write-back through the processor's extended interface. By integrating convolution, activation, and pooling processing into a single hardware design, and combining it with an interface control mechanism adapted to RISC-V processors, intermediate data handling can be reduced, hardware computing unit utilization and system operating efficiency can be improved, thereby better meeting the real-time, low-power, and flexible requirements of embedded intelligent computing scenarios. Summary of the Invention

[0006] To address the issues of excessive intermediate data transfer, inflexible convolutional kernel parameter configuration, and insufficient collaboration efficiency between RISC-V processors and accelerators in existing convolutional neural network (CNN) hardware acceleration solutions, this invention provides a RISC-V processor-based CNN accelerator system. This solution integrates the CNN hardware accelerator structure and its input data into a unified design, enabling integrated processing of convolution, activation, and pooling, and handling the processor's parameter configuration, data transmission, computation initiation, and result return.

[0007] The technical solution of this invention:

[0008] A convolutional neural network accelerator system based on a RISC-V processor is provided to implement convolution computation, activation processing and pooling processing in convolutional neural networks, and to achieve collaborative control between the RISC-V processor and the accelerator.

[0009] The RISC-V processor is a Hummingbird E203 processor, and the convolutional neural network accelerator system is connected to the Hummingbird E203 processor via a NICE interface. The NICE interface is used to enable data interaction between the Hummingbird E203 processor and the convolutional neural network accelerator system, including the transmission of custom extended instructions, memory data, and result data. The convolutional neural network accelerator system includes a weight configuration module, a sliding window generation module, a convolution calculation module, an activation module, a pooling module, a controller, a decoder, a data input module, and a data output module.

[0010] The weight configuration module is connected to the data input module and is used to receive and save the convolution kernel weight data, and to provide the convolution kernel weight data required for convolution calculation to the convolution calculation module.

[0011] The sliding window generation module is connected to the data input module and is used to cache, arrange, and reorganize the input feature map data to generate the local window data required for convolution calculation.

[0012] Furthermore, the sliding window generation module includes a row buffer submodule and a window generation submodule. The row buffer submodule is used to buffer the input feature map data row by row, so that the continuously input input feature map data forms the row alignment relationship required for convolution calculation; the window generation submodule is used to generate local window data required for convolution calculation based on the data output by the row buffer submodule.

[0013] Furthermore, the row buffer submodule employs a three-level first-in-first-out (FIFO) buffer structure, with each level corresponding to a row of input feature map data. After three consecutive rows of input feature map data are input, when subsequent input feature map data continues to be input, the row buffer submodule can simultaneously output the input feature map data from the same column position in three adjacent rows, thereby forming a column of local window data required for convolution calculation.

[0014] Furthermore, the window generation submodule includes multiple storage units, namely nine pseudo-dual-port random access memory (RAMs), each of which includes one write port and one read port. During data writing, the row buffer submodule outputs three data points per clock cycle, and the window generation submodule writes these three data points into the nine pseudo-dual-port RAMs respectively, with each set of three pseudo-dual-port RAMs corresponding to the same row of data. After three consecutive clock cycles of writing, a complete 3×3 local window of data is formed in the nine pseudo-dual-port RAMs. During data reading, by staggering the read addresses of the pseudo-dual-port RAMs, the nine feature values ​​to be calculated for convolution can be read.

[0015] The convolution calculation module is connected to the weight configuration module and the sliding window generation module, respectively, and is used to perform convolution calculation based on local window data and convolution kernel weight data, and output the convolution calculation result.

[0016] Furthermore, the convolution calculation module includes multiple parallel processing elements (PEs). Each processing element (PE) performs a multiplication operation on the feature values ​​in the local window data and the weight values ​​in the corresponding convolution kernel weight data, and accumulates the multiple multiplication results to obtain the convolution calculation result.

[0017] Furthermore, the operations performed by each processing unit (PE) are as follows:

[0018]

[0019] Where F represents the input feature value, W represents the corresponding weight value, C represents the accumulated result passed by the previous processing unit PE, and P represents the partial sum output by the current processing unit PE.

[0020] The convolution calculation module employs a systolic array structure to perform convolution calculations. For a 3×3 convolution kernel, the convolution calculation module includes nine sequentially connected processing units (PEs). The nine weight values ​​transmitted by the weight configuration module are stored in the corresponding processing units (PEs). The nine feature values ​​output by the sliding window generation module are sequentially input into each processing unit (PE). Each processing unit (PE) performs multiplication and addition operations on the corresponding feature value and weight value. The output result of the previous processing unit (PE) is passed to the next processing unit (PE) and multiplied by the next input feature value and the next weight value, thus obtaining the result of one convolution calculation of the 3×3 convolution kernel on the current local window.

[0021] The convolution calculation module includes three parallel convolution calculation paths, each corresponding to one of the three convolution kernels. The same set of nine feature values ​​output by the sliding window generation module are fed into the three convolution calculation paths according to a preset input sequence. First, they are fed into the first processing unit (PE) in each convolution calculation path, and then passed from the previous PE to the next. The weight configuration module provides the corresponding nine weight values ​​for the convolution kernels to each of the three convolution calculation paths, thereby outputting three convolution calculation results in parallel.

[0022] The activation module is connected to the convolution calculation module and is used to perform nonlinear activation processing on the convolution calculation results to obtain feature data after nonlinear activation processing.

[0023] The pooling module is connected to the activation module and the data output module respectively, and is used to perform pooling processing on the feature data after nonlinear activation processing, and output the result data to the data output module.

[0024] Furthermore, the pooling module includes a comparison unit and a caching unit. The comparison unit is used to compare the size of the input feature data after nonlinear activation processing in the row direction, and the caching unit is used to temporarily store and align the data during the pooling process to facilitate the size comparison in the column direction.

[0025] Furthermore, the comparison unit includes a data selector, a register, and a numerical comparator. The data selector selects the data to be temporarily stored and writes it into the register. When the next adjacent feature data is input, the numerical comparator compares the temporarily stored data in the register with the current input data and outputs the larger value as the row direction comparison result. The cache unit adopts a first-in-first-out (FIFO) cache structure to temporarily store the row direction comparison results and align the comparison results of adjacent rows in time. The aligned two row direction comparison results are compared by the numerical comparator to obtain the max pooling result.

[0026] The controller is connected to the decoder, data input module, data output module and NICE interface respectively. It is used to receive instruction information decoded by the decoder, control the data input module and data output module to receive and transmit data, and generate control signals required by the NICE interface.

[0027] Furthermore, the controller includes an interface control unit and a status control unit.

[0028] The interface control unit is used to realize bidirectional control signal communication between the convolutional neural network accelerator system and the Hummingbird E203 processor through the NICE interface. The interface control unit is used to receive control signals sent by the Hummingbird E203 processor in order to respond to and execute custom extended instructions sent by the Hummingbird E203 processor; on the other hand, it is used to send control signals to the Hummingbird E203 processor to request to read memory data or write back result data.

[0029] The state control unit is used to manage and switch the working state of the convolutional neural network accelerator system to coordinate the processes of receiving custom extended instructions, reading memory data, running convolutional calculations, and writing back result data.

[0030] Furthermore, the working states managed by the state control unit include idle state, configuration response state, convolution running state, and running response state. In the idle state, the state control unit allows the interface control unit to receive control signals sent by the Hummingbird E203 processor; in the configuration response state, the state control unit obtains configuration parameters according to the instruction information sent by the decoder and saves them into internal registers; in the convolution running state, the state control unit controls the interface control unit to execute the memory access process in a preset order and drives the data input module to send valid data to the sliding window generation module or the weight configuration module; in the running response state, the interface control unit returns a calculation completion response to the Hummingbird E203 processor through the NICE interface.

[0031] Furthermore, during convolution operation, the interface control unit first initiates a convolution kernel weight data read request based on the base address of the convolution kernel weight data, and sends the returned convolution kernel weight data to the data input module. After the convolution kernel weight data is read, the interface control unit initiates an input feature map data read request based on the base address of the input feature map data, and sends the returned input feature map data to the data input module. After the input feature map data is read and the convolutional neural network accelerator system generates valid result data, the interface control unit initiates a memory write request based on the base address of the valid result data, writing the result data in the data output module back to the corresponding storage space. Once all result data has been written back, the state control unit enters the running response state.

[0032] The decoder connects to the Hummingbird E203 processor via the NICE interface, and is used to identify and decode custom extended instructions sent by the Hummingbird E203 processor, and send the decoded instruction information to the controller. The custom extended instructions are designed for accelerating convolutional neural network tasks, and include instructions for configuring feature map data input / output addresses, configuring the number of feature map data channels, configuring convolution kernel weight data addresses, and initiating convolution computation.

[0033] Furthermore, the feature map data input / output address configuration instruction is used to configure the base address of the input feature map data and the base address of the result data; the feature map data channel number configuration instruction is used to configure the number of input feature map data channels participating in this convolution calculation; the convolution kernel weight data address configuration instruction is used to configure the base address of the convolution kernel weight data; and the start convolution calculation instruction is used to start the convolutional neural network accelerator system to execute the calculation task after the relevant parameters are configured.

[0034] The data input module is connected to the NICE interface, the controller, the sliding window generation module, and the weight configuration module, respectively. Under the control of the controller, it receives memory data transmitted through the NICE interface, converts the received memory data into valid data that adapts to the internal data bit width of the sliding window generation module or the weight configuration module according to the type of the received memory data, and sends the valid data into the corresponding module.

[0035] The data output module is connected to the pooling module, the controller, and the NICE interface respectively; it is used to receive the result data output by the pooling module under the control of the controller, cache the result data, and then write the result data back to the Hummingbird E203 processor through the NICE interface.

[0036] Compared to existing technologies, this invention integrates convolution, activation, and pooling into a single hardware accelerator, thereby reducing the transmission and access of intermediate data between different processing stages and improving the overall processing efficiency of key operators in convolutional neural networks. Simultaneously, this invention includes a weight configuration module, a sliding window generation module, and a convolution calculation module, enabling the accelerator to better adapt to the data organization requirements during convolutional neural network operations and improving the utilization of hardware computing units. Furthermore, this invention introduces a controller and extended instruction set based on the Hummingbird E203 processor's NICE interface, allowing the processor to perform parameter configuration, task initiation, and result return, thus enhancing the collaborative capabilities between general-purpose processors and dedicated accelerators. Attached Figure Description

[0037] Figure 1 This is a diagram of the overall architecture of the accelerator;

[0038] Figure 2 forFigure 1 A schematic diagram of the row buffer structure in the sliding window generation module;

[0039] Figure 3 A flowchart illustrating the module process for generating sliding windows;

[0040] Figure 4 This is a diagram of the convolution computation module architecture.

[0041] Figure 5 This is a flowchart illustrating the pooling module process;

[0042] Figure 6 This is the state transition diagram for the state control unit. Detailed Implementation

[0043] The present invention will be further described below with reference to specific embodiments. It should be noted that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Equivalent substitutions or modifications made by those skilled in the art without departing from the concept of the present invention should fall within the scope of protection of the present invention.

[0044] Example 1:

[0045] like Figure 1 As shown, this embodiment provides a convolutional neural network accelerator system based on a RISC-V processor, which is used to implement convolution calculation, activation processing and pooling processing in convolutional neural networks, and to realize the collaborative control between the RISC-V processor and the accelerator.

[0046] The RISC-V processor is a Hummingbird E203 processor, and the convolutional neural network accelerator system is connected to the Hummingbird E203 processor via a NICE interface. The NICE interface is used to enable data interaction between the Hummingbird E203 processor and the convolutional neural network accelerator system, including the transmission of custom extended instructions, memory data, and result data. The convolutional neural network accelerator system includes a weight configuration module, a sliding window generation module, a convolution calculation module, an activation module, a pooling module, a controller, a decoder, a data input module, and a data output module.

[0047] The weight configuration module is connected to the data input module and is used to receive and save convolution kernel weight data, and provide the convolution kernel weight data required for convolution calculation to the convolution calculation module. In this embodiment, the weight configuration module receives the convolution kernel weight data through the data input module and writes the convolution kernel weight data into an internal register according to a preset writing sequence to form the convolution kernel weight data required by the convolution calculation module. Taking a 3×3 convolution kernel as an example, each convolution kernel corresponds to 9 weight values, and the weight configuration module can simultaneously save the convolution kernel weight data of 3 convolution kernels to support multi-path convolution calculation.

[0048] The sliding window generation module is connected to the data input module and is used to cache, arrange, and reorganize the input feature map data to generate local window data required for convolution calculation. Further, the sliding window generation module includes a row buffer submodule and a window generation submodule.

[0049] The row buffer submodule is used to buffer the input feature map data row by row, so that the continuously input feature map data forms the row alignment relationship required for convolution calculation; preferably, as follows: Figure 2 As shown, the row buffer submodule employs a three-level First-In-First-Out (FIFO) buffer structure, with each level corresponding to a single row of input feature map data. After three consecutive rows of input feature map data are input, when subsequent input feature map data continues, the row buffer submodule can simultaneously output data from the same column position in three adjacent rows, thus forming a column of local window data required for convolution calculation. For example, when the fourth row of input feature map data is input, the row buffer submodule outputs data from the corresponding column positions of the first three rows, thereby achieving row-by-row sliding data output.

[0050] The window generation submodule is used to generate local window data required for convolution calculation based on the data output by the row buffer submodule. Preferably, as follows: Figure 3 As shown, the window generation submodule includes multiple storage units for caching and reading data, thereby providing multiple feature values ​​from the local window data to the convolution calculation module. In this embodiment, the multiple storage units are nine pseudo-dual-port random access memory (RAM), each of which includes a write port and a read port. When writing data, the row buffer submodule outputs three data points per clock cycle, and the window generation submodule writes these three data points into the nine pseudo-dual-port RAMs respectively, with each set of three pseudo-dual-port RAMs corresponding to the same row of data cached. After three consecutive clock cycles of writing, a complete 3×3 local window data is formed in the nine pseudo-dual-port RAMs. When reading data, by staggering the read addresses of each pseudo-dual-port RAM, the nine feature values ​​to be calculated for convolution can be read. Furthermore, the window generation submodule can also perform delay alignment on the nine read feature values ​​to match the data flow timing of the processing unit PE inside the convolution calculation module.

[0051] The convolution calculation module is connected to both the weight configuration module and the sliding window generation module. It performs convolution calculations based on local window data and convolution kernel weight data, and outputs the convolution calculation result. Further, the convolution calculation module includes multiple parallel processing elements (PEs). Each PE performs a multiplication operation on the feature values ​​in the local window data and the weight values ​​in the corresponding convolution kernel weight data, and accumulates the multiple multiplication results to obtain the convolution calculation result.

[0052] In this embodiment, the operation performed by each processing unit PE is as follows:

[0053]

[0054] Where F represents the input feature value, W represents the corresponding weight value, C represents the accumulated result passed by the previous processing unit PE, and P represents the partial sum output by the current processing unit PE.

[0055] Preferably, the convolution calculation module employs a systolic array structure to perform convolution calculations. For a 3×3 convolution kernel, the convolution calculation module includes nine sequentially connected processing units (PEs). The nine weight values ​​transmitted by the weight configuration module are stored in the corresponding processing units (PEs). The nine feature values ​​output by the sliding window generation module are sequentially input into each processing unit (PE). Each processing unit (PE) performs multiplication and addition operations on the corresponding feature value and weight value. The output result of the previous processing unit (PE) is passed to the next processing unit (PE) and multiplied by the next input feature value and the next weight value, thus obtaining the result of one convolution calculation of the 3×3 convolution kernel on the current local window.

[0056] Furthermore, such as Figure 4 As shown, the convolution calculation module includes three sets of parallel convolution calculation paths, each corresponding to one of the three convolution kernels. The same set of nine feature values ​​output by the sliding window generation module are fed into the three convolution calculation paths according to a preset input sequence. First, they are fed into the first processing unit (PE) of each convolution calculation path, and then passed from the previous PE to the next. The weight configuration module provides the corresponding nine weight values ​​for the convolution kernels to each of the three convolution calculation paths, thereby outputting the three convolution calculation results in parallel. With this structure, convolution calculations of three kernels can be completed simultaneously during a single local window input, improving convolution calculation efficiency and hardware resource utilization.

[0057] The activation module is connected to the convolution calculation module and is used to perform nonlinear activation processing on the convolution calculation result to obtain the feature data after nonlinear activation processing. In this embodiment, the activation module adopts the ReLU activation method, that is, it performs sign judgment on the convolution calculation result output by the convolution calculation module. When the convolution calculation result is less than zero, it outputs zero; when the convolution calculation result is greater than or equal to zero, it outputs the original convolution calculation result.

[0058] The pooling module is connected to the activation module and the data output module respectively, and is used to perform pooling processing on the feature data after nonlinear activation processing, and output the result data to the data output module.

[0059] Furthermore, the pooling module includes a comparison unit and a caching unit. The comparison unit is used to compare the size of the input feature data after nonlinear activation processing in the row direction, and the caching unit is used to temporarily store and align the data during the pooling process to facilitate the size comparison in the column direction.

[0060] Preferably, the pooling module employs max pooling, with a pooling window size of 2×2 and a pooling step size of 2. In this embodiment, the pooling process of the pooling module includes two stages: row-wise comparison and column-wise comparison. Figure 5 As shown, in the row-direction comparison stage, the comparison unit compares the size of two adjacent feature data in the same row and outputs the larger value. Further, the comparison unit includes a data selector, a register, and a numerical comparator. The data selector selects data to be temporarily stored and writes it to the register. When the next adjacent feature data is input, the numerical comparator compares the temporarily stored data in the register with the currently input data and outputs the larger value as the row-direction comparison result.

[0061] During the column-direction comparison phase, the pooling module compares the row-direction comparison results at corresponding positions in two adjacent rows to obtain the maximum value within the 2×2 pooling window. Preferably, the cache unit adopts a first-in-first-out (FIFO) cache structure to temporarily store the row-direction comparison results and align the comparison results of adjacent rows in time; the aligned two row-direction comparison results are then compared by a numerical comparator to obtain the max-pooling result. Through the above two-stage processing, the pooling module can complete the max-pooling operation with a pooling window of 2×2 and a pooling step size of 2.

[0062] The controller is connected to the decoder, data input module, data output module and NICE interface respectively. It is used to receive instruction information decoded by the decoder, control the data input module and data output module to receive and transmit data, and generate control signals required by the NICE interface.

[0063] Furthermore, the controller includes an interface control unit and a status control unit.

[0064] The interface control unit is used to realize bidirectional control signal communication between the convolutional neural network accelerator system and the Hummingbird E203 processor through the NICE interface. Specifically, the interface control unit is used to receive control signals sent by the Hummingbird E203 processor to respond to and execute custom extended instructions sent by the Hummingbird E203 processor; and to send control signals to the Hummingbird E203 processor to request reading memory data or writing back result data. Figure 6 As shown, the state control unit is used to manage and switch the operating state of the convolutional neural network accelerator system to coordinate the processes of receiving custom extension instructions, reading memory data, performing convolutional computations, and writing back result data. Further, the operating states managed by the state control unit include idle state, configuration response state, convolution running state, and running response state.

[0065] In idle state, the status control unit allows the interface control unit to receive control signals sent by the Hummingbird E203 processor. When the decoder receives a configuration instruction, the status control unit enters the configuration response state and saves the corresponding configuration parameters according to the instruction information sent by the decoder. After configuration is completed, the interface control unit returns a configuration completion response to the Hummingbird E203 processor through the NICE interface, and the status control unit returns to the idle state.

[0066] Upon receiving a command to initiate convolution computation, the state control unit enters the convolution running state. In this state, the state control unit controls the interface control unit to execute memory access procedures according to a preset sequence. Specifically, the interface control unit first initiates a convolution kernel weight data read request based on the base address of the convolution kernel weight data, and sends the returned convolution kernel weight data to the data input module. After the convolution kernel weight data is read, the interface control unit initiates an input feature map data read request based on the base address of the input feature map data, and sends the returned input feature map data to the data input module. After the input feature map data is read and the convolutional neural network accelerator system generates valid result data, the interface control unit initiates a memory write request based on the base address of the valid result data, writing the result data from the data output module back to the corresponding storage space. Once all result data has been written back, the state control unit enters the running response state.

[0067] Furthermore, the state control unit is also equipped with counting logic to record the completion status of reading convolution kernel weight data, reading input feature map data, and writing back result data, and control the switching between various working states accordingly.

[0068] In the running response state, the interface control unit returns a calculation completion response to the Hummingbird E203 processor via the NICE interface, and the status control unit returns to the idle state to wait for the next control signal.

[0069] The decoder connects to the Hummingbird E203 processor via the NICE interface, and is used to identify and decode custom extended instructions sent by the Hummingbird E203 processor, and send the decoded instruction information to the controller. The custom extended instructions are designed for accelerating convolutional neural network tasks, and include instructions for configuring feature map data input / output addresses, configuring the number of feature map data channels, configuring convolution kernel weight data addresses, and initiating convolution computation.

[0070] Specifically, the feature map data input / output address configuration instruction is used to configure the base address of the input feature map data and the base address of the result data; the feature map data channel number configuration instruction is used to configure the number of input feature map data channels participating in this convolution calculation; the convolution kernel weight data address configuration instruction is used to configure the base address of the convolution kernel weight data; and the start convolution calculation instruction is used to start the convolutional neural network accelerator system to execute the calculation task after the relevant parameters are configured.

[0071] Table 1 illustrates one design approach for the custom extended instruction field in this embodiment.

[0072]

[0073] Preferably, the feature map data input / output address configuration instruction transmits two address parameters simultaneously through a single custom extended instruction. Specifically, the input feature map base address can be transmitted through the rs1 register, and the result data base address can be transmitted through the rs2 register, thereby reducing the number of configuration instructions and improving the configuration efficiency of the Hummingbird E203 processor for the convolutional neural network accelerator system.

[0074] The data input module is connected to the NICE interface, the controller, the sliding window generation module, and the weight configuration module, respectively. Under the control of the controller, it receives memory data transmitted through the NICE interface, converts the received memory data into valid data that adapts to the internal data bit width of the sliding window generation module or the weight configuration module according to the type of the received memory data, and sends the valid data into the corresponding module.

[0075] In this embodiment, the data input module receives 32-bit external data returned via the NICE interface and splits the 32-bit external data into multiple 8-bit data. The split 8-bit data are then sent to the weight configuration module or the sliding window generation module according to the current operating stage of the state control unit. During the weight reading stage, the split 8-bit data is written to the weight configuration module as convolution kernel weight data. During the feature map reading stage, the split 8-bit data is sent to the sliding window generation module as input feature map data to participate in subsequent convolution calculations.

[0076] Furthermore, when the effective data volume of the input convolution kernel weight data or input feature map data is less than one 32-bit transmission unit, the data input module only outputs the effective 8 bits of data and discards the invalid padding data.

[0077] Because the sliding window generation module, convolution calculation module, activation module, and pooling module within the convolutional neural network accelerator system all have a certain pipeline delay, after all input feature map data has been input, the controller can continue to input a predetermined amount of invalid padding data or zero-value data into the convolutional neural network accelerator system through input flushing control logic, while maintaining effective control of the accelerator input. This allows intermediate results that have not yet been output in the pipeline to continue to be passed to subsequent stages until the data output module receives all the valid result data required for this computation task. Using this input flushing control logic avoids the problem of incomplete output of pipeline results due to the end of input feature map data. The data output module is connected to the pooling module, the controller, and the NICE interface; it receives the result data output by the pooling module under the control of the controller, buffers the result data, and then writes the result data back to the Hummingbird E203 processor through the NICE interface.

[0078] In this embodiment, when the pooling module outputs valid result data, the data output module first writes the result data into the output FIFO unit. When the interface control unit can initiate a memory write request, it then reads the result data from the output FIFO unit and writes it back to the storage space corresponding to the result data base address. By using the output FIFO unit, even if the pooling module is outputting result data while the NICE interface is performing other memory read / write operations, the result data can be cached first, thereby alleviating memory read / write conflicts and preventing the loss of valid result data.

[0079] Furthermore, the data output module can merge multiple sets of result data into a single write-back. For example, when the data bit width of a single result data is smaller than the write data bit width of the NICE interface, multiple sets of result data can be concatenated into a 32-bit data and then written back to the external memory in one go.

[0080] To verify the computational performance of the convolutional neural network accelerator system described in this embodiment, the following experimental scenario was set up: the input feature map size was 10×10, the convolution kernel size was 3×3, the number of convolution kernels was 3, the number of input feature map channels was 2, the convolution stride was 1, and 2×2 max pooling was used for pooling.

[0081] This experiment compares two implementation methods. The first is a software implementation, in which the Hummingbird E203 processor executes a C language program to complete convolution calculation, activation processing, and pooling processing. The second is an accelerator implementation, in which the Hummingbird E203 processor completes parameter configuration and computation initiation through custom extended instructions, and the convolutional neural network accelerator system completes convolution calculation, activation processing, and pooling processing.

[0082] The experimental results are shown in the table below:

[0083]

[0084] Compared to the basic C language implementation, the custom instruction set accelerator implementation reduces the number of instructions from 6502 to 14, and the number of execution cycles from 44144 to 251. Therefore, the convolutional neural network accelerator system described in this embodiment can significantly reduce the number of processor instructions executed and the computation cycle, improving the execution efficiency of convolutional neural network operations. In summary, this invention provides a convolutional neural network accelerator system based on a RISC-V processor. The system connects to a Hummingbird E203 processor via a NICE interface and utilizes custom extended instructions to complete parameter configuration, computation initiation, and result write-back. Under the coordination of the controller, the data input module, weight configuration module, sliding window generation module, convolution calculation module, activation module, pooling module, and data output module sequentially complete data reading, weight configuration, convolution calculation, activation processing, pooling processing, and result output. Thus, this invention can reduce the number of processor instructions executed and data transport overhead, improve the execution efficiency of convolutional neural network operations, and ensure system stability.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent substitutions, modifications, or variations made based on the concept of the present invention should be included within the scope of protection of the present invention.

Claims

1. A convolutional neural network accelerator system based on a RISC-V processor, characterized in that, It is used to implement convolution calculation, activation processing and pooling processing in convolutional neural networks, and to realize the cooperative control between RISC-V processors and accelerators; The RISC-V processor is a Hummingbird E203 processor, and the convolutional neural network accelerator system is connected to the Hummingbird E203 processor via a NICE interface. The NICE interface is used to realize data interaction between the Hummingbird E203 processor and the convolutional neural network accelerator system, and the data interaction includes the transmission of custom extended instructions, memory data, and result data. The convolutional neural network accelerator system includes a weight configuration module, a sliding window generation module, a convolution calculation module, an activation module, a pooling module, a controller, a decoder, a data input module, and a data output module. The weight configuration module is connected to the data input module and is used to receive and save the convolution kernel weight data, and to provide the convolution kernel weight data required for convolution calculation to the convolution calculation module. The sliding window generation module is connected to the data input module and is used to cache, arrange and reorganize the input feature map data to generate local window data required for convolution calculation. The convolution calculation module is connected to the weight configuration module and the sliding window generation module respectively, and is used to perform convolution calculation based on local window data and convolution kernel weight data, and output the convolution calculation result; The activation module is connected to the convolution calculation module and is used to perform non-linear activation processing on the convolution calculation results to obtain feature data after non-linear activation processing. The pooling module is connected to the activation module and the data output module respectively, and is used to perform pooling processing on the feature data after nonlinear activation processing, and output the result data to the data output module. The controller is connected to the decoder, data input module, data output module and NICE interface respectively. It is used to receive instruction information decoded by the decoder, control the data input module and data output module to receive and transmit data, and generate control signals required by the NICE interface. The decoder connects to the Hummingbird E203 processor via the NICE interface and is used to identify and decode custom extended instructions sent by the Hummingbird E203 processor, and send the decoded instruction information to the controller. The custom extended instructions are designed for convolutional neural network acceleration tasks and include feature map data input / output address configuration instructions, feature map data channel number configuration instructions, convolution kernel weight data address configuration instructions, and convolution calculation start instructions. The data input module is connected to the NICE interface, the controller, the sliding window generation module, and the weight configuration module, respectively. Under the control of the controller, it receives memory data transmitted through the NICE interface, converts the received memory data into valid data that adapts to the internal data bit width of the sliding window generation module or the weight configuration module according to the type of the received memory data, and sends the valid data into the corresponding module. The data output module is connected to the pooling module, the controller, and the NICE interface respectively; it is used to receive the result data output by the pooling module under the control of the controller, cache the result data, and then write the result data back to the Hummingbird E203 processor through the NICE interface. The sliding window generation module includes a row buffer submodule and a window generation submodule; the row buffer submodule is used to buffer the input feature map data row by row, so that the continuously input input feature map data forms the row alignment relationship required for convolution calculation; the window generation submodule is used to generate local window data required for convolution calculation based on the data output by the row buffer submodule. The row buffer submodule adopts a three-level first-in-first-out (FIFO) buffer structure, with each level of the FIFO buffer structure corresponding to a row of input feature map data. When three rows of input feature map data are input consecutively, the row buffer submodule can simultaneously output the input feature map data of the same column position in three adjacent rows when subsequent input feature map data are input, thereby forming a column of local window data required for convolution calculation. The window generation submodule includes multiple storage units, which are nine pseudo-dual-port random access memory (RAMs). Each pseudo-dual-port RAM includes one write port and one read port. When writing data, the row buffer submodule outputs three data points in each clock cycle, and the window generation submodule writes these three data points into the nine pseudo-dual-port RAMs respectively, with each set of three pseudo-dual-port RAMs corresponding to the same row of data. After three consecutive clock cycles of writing, a complete 3×3 local window data is formed in the nine pseudo-dual-port RAMs. When reading data, the nine feature values ​​to be calculated for convolution are read out by staggering the read addresses of each pseudo-dual-port RAM. The pooling module includes a comparison unit and a caching unit. The comparison unit is used to compare the size of the input feature data after nonlinear activation processing in the row direction, and the caching unit is used to temporarily store and align the data during the pooling process to facilitate the size comparison in the column direction. The comparison unit includes a data selector, a register, and a numerical comparator. The data selector selects the data to be temporarily stored and writes it into the register. When the next adjacent feature data is input, the numerical comparator compares the temporarily stored data in the register with the current input data and outputs the larger value as the row direction comparison result. The cache unit adopts a first-in-first-out (FIFO) cache structure to temporarily store the row direction comparison results and align the comparison results of adjacent rows in time. The aligned two row direction comparison results are compared by the numerical comparator to obtain the max pooling result.

2. The convolutional neural network accelerator system based on a RISC-V processor according to claim 1, characterized in that, The convolution calculation module includes multiple parallel processing units (PEs). Each processing unit (PE) performs a multiplication operation on the feature values ​​in the local window data and the weight values ​​in the corresponding convolution kernel weight data, and accumulates the multiple multiplication results to obtain the convolution calculation result. The operations performed by each processing unit (PE) are as follows: Where F represents the input feature value, W represents the corresponding weight value, C represents the accumulated result passed by the previous processing unit PE, and P represents the partial sum output by the current processing unit PE.

3. The convolutional neural network accelerator system based on a RISC-V processor according to claim 2, characterized in that, The convolution calculation module uses a pulsating array structure to perform convolution calculations. For a 3×3 convolution kernel, the convolution calculation module includes nine sequentially connected processing units (PEs). The nine weight values ​​transmitted by the weight configuration module are stored in the corresponding processing units (PEs). The nine feature values ​​output by the sliding window generation module are sequentially input into each processing unit (PE). Each processing unit (PE) performs multiplication and addition operations on the corresponding feature value and weight value. The output result of the previous processing unit (PE) is passed to the next processing unit (PE) and multiplied by the next input feature value and the next weight value, thus obtaining the result of one convolution calculation of the 3×3 convolution kernel on the current local window. The convolution calculation module includes three sets of parallel convolution calculation paths, each corresponding to a convolution kernel. The same set of nine feature values ​​output by the sliding window generation module are sent to the three sets of convolution calculation paths according to a preset input timing sequence. They are first sent to the first column processing unit (PE) in each convolution calculation path, and then passed from the previous column processing unit (PE) to the next column processing unit (PE). The weight configuration module provides the nine weight values ​​corresponding to the convolution kernel to the three sets of convolution calculation paths, thereby outputting three convolution calculation results in parallel.

4. The convolutional neural network accelerator system based on a RISC-V processor according to claim 1, characterized in that, The controller includes an interface control unit and a status control unit; The interface control unit is used to realize bidirectional control signal communication between the convolutional neural network accelerator system and the Hummingbird E203 processor through the NICE interface. The interface control unit is used to receive control signals sent by the Hummingbird E203 processor in order to respond to and execute the custom extended instructions sent by the Hummingbird E203 processor; on the other hand, it is used to send control signals to the Hummingbird E203 processor to request to read memory data or write back result data. The state control unit is used to manage and switch the working state of the convolutional neural network accelerator system to coordinate the processes of receiving custom extended instructions, reading memory data, running convolutional calculations, and writing back result data. The state control unit manages the following working states: idle state, configuration response state, convolution running state, and running response state. In the idle state, the state control unit allows the interface control unit to receive control signals sent by the Hummingbird E203 processor. In the configuration response state, the state control unit obtains configuration parameters according to the instruction information sent by the decoder and saves them to the internal register. In the convolution running state, the state control unit controls the interface control unit to execute the memory access process in a preset order and drives the data input module to send valid data to the sliding window generation module or the weight configuration module. In the running response state, the interface control unit returns a calculation completion response to the Hummingbird E203 processor through the NICE interface.

5. The convolutional neural network accelerator system based on a RISC-V processor according to claim 4, characterized in that, During convolution operation, the interface control unit first initiates a convolution kernel weight data read request based on the base address of the convolution kernel weight data, and then sends the returned convolution kernel weight data into the data input module. After the convolution kernel weight data is read, the interface control unit initiates an input feature map data reading request based on the base address of the input feature map data, and sends the returned input feature map data into the data input module; After the input feature map data is read and the convolutional neural network accelerator system generates valid result data, the interface control unit initiates a memory write request based on the base address of the valid result data to write the result data in the data output module back to the corresponding storage space. Once all the result data has been written back, the status control unit enters the running response state.

6. The convolutional neural network accelerator system based on a RISC-V processor according to claim 1, characterized in that, The feature map data input / output address configuration instruction is used to configure the base address of the input feature map data and the base address of the result data; the feature map data channel number configuration instruction is used to configure the number of input feature map data channels participating in this convolution calculation; the convolution kernel weight data address configuration instruction is used to configure the base address of the convolution kernel weight data; the start convolution calculation instruction is used to start the convolutional neural network accelerator system to execute the calculation task after the relevant parameters are configured.

Images