RISC-v based neural network coprocessor control method and coprocessor architecture

Abstract

This invention discloses a RISC-V-based neural network coprocessor control method and coprocessor architecture. The coprocessor processes image data in parallel and processes the corresponding output rows in a fixed manner, eliminating the redundancy of Im2Col data rearrangement. Combined with in-situ accumulation using a local circular buffer, it reduces memory bandwidth and storage waste. A decentralized distributed micro-caching architecture is adopted, configuring a small-capacity private local circular buffer for the coprocessor to reduce SRAM area and interconnect logic overhead. A shadow weight storage area is set up to provide weights in parallel and reusable modes in convolution and fully connected modes, breaking through the SRAM bandwidth limitation and solving the bottleneck problems of address wall, memory wall and fully connected layer. Through a hybrid mode state machine, the data path is dynamically reconstructed. In convolution mode, convolution, accumulation, activation and pooling operations can be executed continuously. In fully connected mode, the computing power of the convolution array is reused to perform vector dot product. Without increasing the number of multipliers, it breaks through the fixed structure limitation of traditional accelerators and further improves the inference efficiency of neural networks.

Description

Technical Field

[0001] This invention relates to the field of neural network acceleration processor technology, and in particular to a RISC-V-based neural network coprocessor control method and coprocessor architecture. Background Technology

[0002] With the popularization of deep learning, the demand for neural network inference in edge devices (smart sensors, wearable devices, etc.) has exploded (i.e. TinyML). However, edge devices have stringent requirements for chip area, power consumption, and real-time performance, which existing technologies cannot adapt to.

[0003] There are three main types of existing edge-side convolutional neural network deployment methods, all of which have obvious drawbacks:

[0004] General-purpose processors (such as Cortex-M and standard RISC-V cores) perform matrix operations by executing software instruction sequences, which has the following drawbacks: Each multiplication-accumulation operation in a convolutional neural network requires a complete pipeline stage of instruction fetch, decoding, execution, and memory access, resulting in a large number of clock cycles being consumed in instruction control flow, high instruction overhead, and the address wall problem; In memory-intensive operators such as fully connected layers, the processor needs to frequently read weights and activation values ​​from DRAM, while the access latency of main memory (usually tens to hundreds of clock cycles) is much higher than the computation latency (multiplication-accumulation operations only require 1-2 cycles), causing arithmetic units to be idle for a large amount of time, resulting in limited memory bandwidth and the memory wall problem.

[0005] Traditional systolic array accelerators (such as the miniaturized Google TPU) rely on the systolic transfer of data between processing units to reuse data, which has the following drawbacks: When implementing convolutional neural network acceleration, in order to maintain the high utilization of the array, a huge unified SRAM buffer must be integrated on the chip, resulting in large area overhead; Since it is essentially a matrix multiplication accelerator, calculating convolution requires performing Im2Col data transformation first, which causes the amount of input data to expand (for example, 3x3 convolution will result in 9 times data redundancy), which greatly squeezes memory bandwidth and storage space.

[0006] The drawback of lightweight dedicated accelerators is that they only optimize convolutional layers. When implementing fully connected layers, the computation mode changes to vector-matrix multiplication, and there is a lack of opportunities to reuse weight parameters. As a result, accelerators often do not support or are extremely inefficient, and they have to fall back to CPU execution, causing the "Amdahl bottleneck" problem in fully connected layers.

[0007] The RISC-V instruction set architecture has the advantages of modularity and scalability. Its basic instructions are simplified and reserve sufficient space for custom coding, which allows hardware accelerators to be deeply coupled with the main core in the form of coprocessors without breaking ISA compatibility. It is an ideal basis for custom dedicated coprocessors.

[0008] Given the low energy efficiency issues in existing neural network inference technologies, such as address walls, memory walls, large area overhead, wasted memory bandwidth and storage space, and the "Amdahl bottleneck" in fully connected layers, it is necessary to propose a RISC-V-based neural network coprocessor control method and coprocessor architecture to solve the above-mentioned technical problems. Summary of the Invention

[0009] This invention provides a RISC-V-based neural network coprocessor control method and coprocessor architecture, which solves the problem of low energy efficiency in neural network inference in existing technologies.

[0010] The present invention solves the technical problem through the following technical solution:

[0011] The RISC-V-based neural network coprocessor control method includes the following steps:

[0012] (1) Define access instructions, compound operation instructions and loading instructions based on RISC-V architecture; load external pre-trained weight parameters and image data to be inferred;

[0013] (2) Allocate a local circular buffer and a shadow weight storage area; the local buffer space is used to cache the data to be processed and the calculation results, and the shadow weight storage area is used to cache the weight parameters to be called;

[0014] (3) Combining the pre-trained weight parameters, the image data to be inferred is processed in parallel on a row-by-row basis. For each row of data in the image to be inferred, the processing steps are as follows:

[0015] (31) Execute the access instruction, preset the convolution window, and load the data of the convolution window size and the corresponding pre-trained weight parameters into the local circular buffer;

[0016] (32) Execute the loading instruction to load the pre-trained weight parameters cached in the local circular buffer into the shadow weight storage area for subsequent calculations;

[0017] (33) Execute composite operation instructions and perform corresponding operations according to the operation mode: For convolution mode, perform one or more of the operations of convolution, accumulation, activation and pooling according to the parameters provided by the composite operation instructions; For fully connected mode, for each image category, reload the pre-trained weight parameters corresponding to the category into the weight parameter storage area of ​​the local circular buffer, and then perform vector dot product operations on each image category in a single-core serial multiplexing manner.

[0018] Furthermore, in step (3), a number of coprocessors greater than or equal to the total number of rows of the image data to be inferred are configured, and processors in an idle state are scheduled to execute steps (31), (32), and (33) in parallel on a row-by-row basis to achieve parallel processing of multiple rows of data.

[0019] Further, in step (31), the local circular buffer is configured with the following storage areas:

[0020] The data storage area is used to store the data to be processed corresponding to the current convolution window;

[0021] The results storage area is used to store the operation results and intermediate iteration results in convolution mode and fully connected mode;

[0022] The weight parameter storage area is used to store the pre-trained weight parameters that correspond one-to-one with each image category.

[0023] Furthermore, in the convolution mode of step (33), the specific implementation of each operation is as follows:

[0024] The convolution operation involves multiplying the data to be processed in the local circular buffer and the corresponding weight parameters in the shadow weight register using a preset convolution kernel, and then summing the results of each multiplication operation to generate an initial convolution response value.

[0025] The accumulation operation adds the initial convolution response value generated this time to the previous convolution response value stored in the result storage area of ​​the local circular buffer, updates the response value, and stores it back in the result storage area;

[0026] The activation operation involves checking the sign bit of the accumulated response value. If a negative response value is detected, the response value is forcibly set to zero, and a non-negative response value is retained as the activation result. Otherwise, the response value is retained as the activation result.

[0027] Pooling operation extracts the maximum value among all data within a preset pooling window, using it as the feature value of the current pooling window data, and stores it in the result storage area.

[0028] Furthermore, in the fully connected mode of step (33), the implementation steps of the single-core serial multiplexing are as follows:

[0029] (331) Activate one of the idle coprocessors as a busy coprocessor, and activate one of the multiplication functions of the busy coprocessor to specifically perform vector dot product operations in fully connected mode;

[0030] (332) Cut off the functions of accumulation, activation and pooling operations in the busy coprocessor, and retain only the convolution operation function. Unfold the convolution operation result and load it into the result storage area provided by the busy coprocessor. At the same time, load the pre-trained weight parameters corresponding to the convolution operation result into the weight parameter storage area of ​​the local circular buffer of the busy coprocessor.

[0031] (333) The convolution operation unrolling result and the corresponding pre-trained weight parameters loaded in step (332) will reside in the corresponding data storage area throughout all vector dot product operations in the subsequent fully connected mode, and will not be loaded again. The busy coprocessor performs vector dot product operations on the loaded convolution operation unrolling result and the corresponding pre-trained weight parameters. After completing the operation for one image category, it switches to the next image category and repeats the above vector dot product operation process until all image categories have been operated.

[0032] A coprocessor architecture for executing the aforementioned RISC-V-based neural network coprocessor control method:

[0033] It includes an instruction distributor and at least one coprocessor; the number of coprocessors is greater than or equal to the total number of rows of the image data to be inferred; the instruction distributor is electrically connected to each coprocessor and is used to schedule the working state of each coprocessor.

[0034] When each coprocessor is idle, the image data to be inferred is processed in parallel on a line-by-line basis under the allocation and scheduling of the instruction allocator.

[0035] The instruction distributor is equipped with an instruction sliding buffer unit, which is signal-connected to each coprocessor. The instruction sliding buffer unit is used to temporarily store instructions to be sent to the target coprocessor when the target coprocessor is busy, and to send the cached instructions to the coprocessor when the target coprocessor is idle. Each coprocessor interacts with the instruction distributor to receive instructions and perform the arithmetic processing described in steps (31), (32), and (33).

[0036] Furthermore, each coprocessor includes a distributed memory unit, a shadow weight register set, an arithmetic logic unit, and a hybrid-mode state machine, with signal interaction between the components;

[0037] The distributed storage unit has a built-in local circular buffer, which interacts with the shadow weight register signal to store the row data to be processed, weight parameters, and calculation results.

[0038] The shadow weight register group interacts with the arithmetic logic unit via signals. The shadow weight register group provides a shadow weight storage area for storing the pre-trained weight data required for the current operation of the arithmetic logic unit.

[0039] The arithmetic logic unit is connected to the hybrid mode state machine signal. The arithmetic logic unit integrates various operation modules, including: a convolution kernel operator for performing convolution operations, a channel accumulator for performing multi-channel partial summation, an activation function module for correcting data signs, and a max pooling logic module for filtering and compressing data.

[0040] The hybrid-mode state machine is used to switch the coprocessor's operation mode, including convolution mode and fully connected mode, and is used to output a mode selection signal to the arithmetic logic unit. The arithmetic logic unit performs the operation in the corresponding mode according to the mode selection signal.

[0041] Furthermore, the convolution kernel operator includes nine 32-bit multipliers and one adder tree; all nine 32-bit multipliers are signal-connected to the shadow weight register group and the data storage area of ​​the local circular buffer, and are also signal-connected to the adder tree.

[0042] Each multiplier operates in parallel, receiving weight data from the shadow weight register and data of the convolution window size from the data storage area of ​​the local circular buffer, and synchronously performing element-wise dot product operations; the addition tree sums the results of each dot product operation and outputs the initial response value of the convolution operation.

[0043] Furthermore, the activation function module includes a sign bit comparator and a multiplexer; the sign bit comparator is connected to the channel accumulator signal, and the multiplexer is connected to the sign bit comparator and the max pooling logic module signal respectively;

[0044] The sign bit comparator is used to detect the sign bit of the operation result output by the channel accumulator, determine whether the operation result is positive or negative, and output a judgment signal to the multiplexer.

[0045] The multiplexer performs a gating operation based on the judgment signal: when the operation result is positive, it selects and outputs the operation result; when the operation result is negative, it selects the zero signal, forces the operation result to zero and outputs it.

[0046] Furthermore, the max pooling logic module includes a horizontal comparator and a vertical comparator; the horizontal comparator is connected to the pooling temporary storage area of ​​the local circular buffer, and the vertical comparator is connected to the horizontal comparator and the result storage area of ​​the local circular buffer, respectively.

[0047] The horizontal comparator performs pairwise comparisons on each row of data within the preset pooling window, extracts the maximum value in each row, and sends it to the vertical comparator. The vertical comparator then compares the maximum values ​​extracted by the horizontal comparator on each row again, and finally extracts the maximum value of the data within the entire pooling window, which is then output as the feature value of the pooling window to the result storage area of ​​the local circular buffer.

[0048] The advantages and effects of this invention are:

[0049] The coprocessor in the idle state processes the image data to be inferred in parallel on a row-by-row basis. Each coprocessor processes a specific output row, eliminating the 9-fold data redundancy caused by Im2Col data rearrangement. Combined with the in-situ accumulation mechanism of the local circular buffer, it avoids repeated reading and writing of intermediate results. This invention works by combining row resident and row parallelism, effectively solving the problem of wasted memory bandwidth and storage space.

[0050] A very small private local circular buffer is allocated to each coprocessor to store the current data of the size of the convolution window obtained from main memory (DRAM), the corresponding pre-trained weight parameters, and the subsequent computation results. There is no data exchange network between the coprocessors. This decentralized distributed micro-cache architecture effectively reduces the SRAM area, trades space for time, greatly simplifies the interconnection logic, and effectively solves the problem of large area overhead.

[0051] The shadow weight storage area is allocated to the coprocessor, which can provide weight parameters in parallel in both convolution mode and fully connected mode. This breaks through the read bandwidth limitation of single-port SRAM, provides an efficient data supply path for composite operations, and can be continuously reused in subsequent composite operations after loading, without having to read it again for each operation. This effectively solves the problems of address wall, memory wall and "Amdahl bottleneck" in fully connected layer.

[0052] A hybrid mode state machine is defined that can distinguish between convolutional and fully connected modes. In convolutional mode, one or more operations such as convolution, accumulation, activation, and pooling are performed without intermediate storage. In fully connected mode, the fixed concatenation of the convolutional array is automatically cut off, the data path is locked, and the computing resources of the convolutional kernel are reused to perform a loop traversal in the local circular buffer to perform vector dot product. The reuse of the data path is achieved by exchanging minimal control logic overhead. The above-mentioned dynamic reconstruction of the data path breaks through the physical limitation of traditional hardware accelerators that can only perform fixed convolutional microstructures without introducing large-scale multipliers. It further solves the address wall, memory wall, and "Amdahl bottleneck" problem of fully connected layers. Attached Figure Description

[0053] Figure 1 This invention relates to a coprocessor control method.

[0054] Figure 2 This is a structural block diagram of the main processor and coprocessor architecture of the present invention.

[0055] Figure 3 This is the UML diagram of the hybrid mode state machine of this invention.

[0056] Figure 4 This is the data flow during the execution process of this invention. Detailed Implementation

[0057] The present invention will be further described below with reference to the embodiments, but the present invention is not limited to these embodiments.

[0058] The RISC-V-based neural network coprocessor control method includes the following steps:

[0059] (1) Define access instructions, compound operation instructions and loading instructions based on RISC-V architecture; load external pre-trained weight parameters and image data to be inferred;

[0060] (2) Allocate a local circular buffer and a shadow weight storage area; the local buffer space is used to cache the data to be processed and the calculation results, and the shadow weight storage area is used to cache the weight parameters to be called;

[0061] (3) Combining the pre-trained weight parameters, the image data to be inferred is processed in parallel on a row-by-row basis. For each row of data in the image to be inferred, the processing steps are as follows:

[0062] (31) Execute the access instruction, preset the convolution window, and load the data of the convolution window size and the corresponding pre-trained weight parameters into the local circular buffer;

[0063] (32) Execute the loading instruction to load the pre-trained weight parameters cached in the local circular buffer into the shadow weight storage area for subsequent calculations;

[0064] (33) Execute composite operation instructions and perform corresponding operations according to the operation mode: For convolution mode, perform one or more of the operations of convolution, accumulation, activation and pooling according to the parameters provided by the composite operation instructions; For fully connected mode, for each image category, reload the pre-trained weight parameters corresponding to the category into the weight parameter storage area of ​​the local circular buffer, and then perform vector dot product operations on each image category in a single-core serial multiplexing manner.

[0065] The structural block diagram of the main processor and coprocessor architecture of this invention is as follows: Figure 2As shown. The RISC-V instruction set architecture extends a custom instruction set, using the CUSTOM-0 opcode space. The main processor (CPU core) defines the RS.DMA access instruction, RS.CALC compound arithmetic instruction, and RS.WLOAD load instruction based on the RISC-V architecture. The RS instructions are compiled into machine code, which includes coprocessor identifier bits and function parameter bits. After passing through the address fetch stage (IF), decode stage (ID), and execution stage (EX), the machine code is routed to the target coprocessor for parallel computation via the instruction dispatcher. After the computation is completed, the result is fed back to the CPU core through the memory access stage (MEM) and the write-back stage (WB).

[0066] Furthermore, the encoding format of the access instruction RS.DMA is shown in Table 1:

[0067] Table 1 Encoding format of the RS.DMA access command

[0068]

[0069] The encoding format of the compound operation instruction RS.CALC is shown in Table 2:

[0070] Table 2 Encoding format of compound arithmetic instruction RS.CALC

[0071]

[0072] The encoding format of the load command RS.WLOAD is shown in Table 3:

[0073] Table 3 Encoding format of the load command RS.WLOAD

[0074]

[0075] During the address fetch phase (IF), the CPU core reads the 32-bit machine code from the program memory. During the decoding phase (ID), it checks the 0-6 bits of the instruction's opcode. When it detects 0001011, it identifies it as an RS user-defined instruction. Based on the 14-12 bits of the instruction's funct3 field, it distinguishes the specific instruction type: if funct3 == 100, it is identified as an RS.DMA instruction; if funct3 == 010, it is identified as an RS.CALC instruction; if funct3 == 011 (binary 011), it is identified as an RS.WLOAD instruction. Based on the content of rs2, it reads the 32-bit value of the x[rs2] register from the general-purpose register file, parses the content according to the instruction definition table, and sends the instruction information to the instruction dispatcher.

[0076] Figure 2The on-chip design of the CNN accelerator is the coprocessor architecture of the present invention, used to execute the above-mentioned RISC-V-based neural network coprocessor control method. The on-chip design of the CNN accelerator mainly includes an instruction dispatcher and a coprocessor array, wherein the coprocessor array is configured with n (n is natural) coprocessors, and n is greater than or equal to the total number of rows of image data to be inferred. The instruction dispatcher schedules the processors in the idle state to execute steps (31), (32), and (33) in parallel on a row-by-row basis, realizing the parallel processing of multiple rows of data. After each coprocessor completes its processing, it forms a completion queue and returns to the main processor.

[0077] A coprocessor architecture includes an instruction distributor and at least one coprocessor; the number of coprocessors is greater than or equal to the total number of rows of the image data to be inferred; the instruction distributor is electrically connected to each coprocessor and is used to schedule the working state of each coprocessor.

[0078] When each coprocessor is idle, the image data to be inferred is processed in parallel on a line-by-line basis under the allocation and scheduling of the instruction allocator.

[0079] The instruction distributor is equipped with an instruction sliding buffer unit, which is signal-connected to each coprocessor. The instruction sliding buffer unit is used to temporarily store instructions to be sent to the target coprocessor when the target coprocessor is busy, and to send the cached instructions to the coprocessor when the target coprocessor is idle. Each coprocessor interacts with the instruction distributor to receive instructions and perform the arithmetic processing described in steps (31), (32), and (33).

[0080] The instruction dispatcher's skid buffer caches current instructions when the coprocessor is busy, allowing the CPU core's pipeline to proceed without waiting. The specific dispatch process is as follows:

[0081] After receiving the RS instruction, the instruction dispatcher checks the busy status of the target coprocessor;

[0082] If the target coprocessor is idle, the instruction is directly passed to the target coprocessor;

[0083] If the target coprocessor is busy and the Skid Buffer is empty (not occupied), the instruction is latched into the Skid Buffer, the CPU core considers the instruction to be successfully issued, and continues to execute the next instruction, thus hiding the coprocessor's waiting delay;

[0084] If the target coprocessor is busy and the Skid Buffer is full (already occupied), the dispatcher sends a pause signal to the CPU pipeline until the coprocessor becomes idle and finishes executing the previous task, freeing up space in the local circular buffer.

[0085] The coprocessor control method of the present invention is as follows: Figure 1 As shown. The data interaction between the instruction dispatcher and the target coprocessor is a control flow and state handshake interaction, and the specific data exchanged includes:

[0086] (a) Downlink control parameters (instruction dispatcher sends to target coprocessor): The instruction configuration interface through which the instruction dispatcher extracts the effective instruction payload after the CPU core is decoded and sends it to the target coprocessor. It mainly includes: operation control flags (such as activation enable relu_en, pooling enable pooling_en, accumulation enable acc_en, calculation mode selection fc_mode / is_wload, pooling phase pool_phase), the offset address and length definition of the internal cache buffer (buf_ptr, wbuf_offset, count_words), the DRAM physical base address (for RS.DMA instructions, the 32-bit base address value issued by the main processor source register is directly transferred), and the data transfer direction flag (is_write).

[0087] (b) Uplink Status Signals (Feedback from the Target Coprocessor to the Target Co-distributor): The instruction configuration interface of the target coprocessor feeds back pure hardware-level status flags to the instruction distributor in real time, mainly coprocessor status handshake signals (Ready or Busy). The instruction distributor relies heavily on this uplink feedback signal to determine the corresponding instruction routing strategy (directly pass-through new instructions, push them into the Skid Buffer, or block the CPU decoding pipeline).

[0088] Each coprocessor includes a distributed memory unit, a shadow weight register group, an arithmetic logic unit, and a hybrid mode state machine, with signal interaction between the components;

[0089] The distributed storage unit has a built-in local circular buffer, which interacts with the shadow weight register signal to store the row data to be processed, weight parameters, and calculation results.

[0090] The shadow weight register group interacts with the arithmetic logic unit via signals. The shadow weight register group provides a shadow weight storage area for storing the pre-trained weight data required for the current operation of the arithmetic logic unit.

[0091] The arithmetic logic unit is connected to the hybrid mode state machine signal. The arithmetic logic unit integrates various operation modules, including: a convolution kernel operator for performing convolution operations, a channel accumulator for performing multi-channel partial summation, an activation function module for correcting data signs, and a max pooling logic module for filtering and compressing data.

[0092] The hybrid-mode state machine is used to switch the coprocessor's operation mode, including convolution mode and fully connected mode, and is used to output a mode selection signal to the arithmetic logic unit. The arithmetic logic unit performs the operation in the corresponding mode according to the mode selection signal.

[0093] In step (31), the local circular buffer is configured with the following storage areas:

[0094] The data storage area is used to store the data to be processed corresponding to the current convolution window;

[0095] The results storage area is used to store the operation results and intermediate iteration results in convolution mode and fully connected mode;

[0096] The weight parameter storage area is used to store the pre-trained weight parameters that correspond one-to-one with each image category.

[0097] The distributed storage unit includes a local circular buffer with a depth of 128 bits and a width of 32 bits, and its address mapping is shown in Table 4:

[0098] Table 4. Mapping Relationship of Distributed Storage Units

[0099]

[0100] The shadow weight register group is a register array located between the distributed storage unit and the arithmetic logic unit, providing a shadow weight storage area. The shadow weight register group contains 16 independent 32-bit registers (index 0-15), where indices 0-8 correspond to the 9 weight parameters w[0][0] to w[2][2] of the 3x3 convolution kernel, respectively. Indices 0-15 can be used as a weight vector register group of length 16 in fully connected mode to provide counter traversal reading. Implemented by flip-flops, it supports 9-way parallel read operations, providing weight data to 9 downstream multipliers in a single clock cycle, breaking through the physical limitation that the distributed storage unit can only provide a single-cycle single read port, and providing an efficient data supply path for convolution operations; triggered by the RS.WLOAD instruction, the 9 32-bit data of the weight parameter storage area (weight buffer, starting from index 80) of the distributed storage unit are latched in parallel to the shadow weight register group by a dedicated data path. After loading, it is continuously reused in subsequent convolution calculations without having to read the weight parameters again for each calculation.

[0101] The Arithmetic Logic Unit (ALU) employs an operator fusion design, cascading convolution, accumulation, activation, and pooling into a tightly coupled combinational logic path. The coprocessor's core idea is to construct this tightly coupled combinational logic path, enabling flexible concatenation of multiple operations based on instruction function parameters (such as relu_en, pooling_en, and acc_en in RS.CALC). This avoids repeatedly writing intermediate calculation results to main memory and then reading them back, significantly reducing system bandwidth consumption. RS.CALC custom instructions can control the system to complete the entire "convolution + accumulation + activation + pooling" calculation process, eliminating the bandwidth overhead of writing intermediate results back to memory.

[0102] The arithmetic logic unit contains four cascaded arithmetic modules:

[0103] (a') Convolution Kernel: The convolution kernel operator simulates the process of human eyes observing objects. It uses a multiplier array to perform a weighted sum of local window pixels of the image with the weights of the convolution kernel, thereby extracting features such as image edges or textures.

[0104] The convolution kernel operator includes nine 32-bit multipliers and one adder tree. Each of the nine 32-bit multipliers is signal-connected to the shadow weight register group, the local circular buffer's data storage area, and the adder tree. Each multiplier operates in parallel, receiving weight data from the shadow weight register and the convolution window size data from the local circular buffer's data storage area, and synchronously performing element-wise dot product operations. The adder tree sums the results of each dot product operation and outputs the initial response value of the convolution operation.

[0105] (b') Channel Accumulator: Located before the ReLU activation function, neural networks are typically composed of dozens or even hundreds of feature channels stacked together (e.g., an RGB image has 3 channels). It adds the currently calculated initial convolution response value to the partial sum already accumulated in the buffer by establishing a readback path. It is used to process multi-layered image features.

[0106] During the intermediate steps of multi-channel convolution, the control parameters acc_en=1 (enable accumulation) and relu_en=0 (disable activation) are used. At this time, the channel accumulator reads back the intermediate sum that was not activated in the previous round, adds it to the current convolution result, and directly passes through the subsequently disabled ReLU and pooling units, writing it back to the buffer without loss, thus ensuring the linearity of the accumulation process. Only when calculating the last channel is acc_en=1 and relu_en=1 enabled at the same time to complete the final accumulation, activation and pooling output.

[0107] (c') The ReLU Unit (Activation Function Module) simulates the "all or nothing" characteristic of biological neurons. It uses a sign bit comparator and a multiplexer to determine the sign of the data. If the value is negative, it outputs zero; otherwise, it retains a positive value. This process eliminates redundant information and introduces a non-linear convolution operator output. The ReLU Unit consists of a sign bit comparator and a multiplexer. When the instruction control bit `relu_en=1`, if the highest bit (sign bit) of the convolution result is 1, the output is forced to 0; otherwise, the original value is retained. This achieves a zero-overhead ReLU(x) = Max(0, x) operation through hardware wiring.

[0108] The activation function module includes a sign bit comparator and a multiplexer; the sign bit comparator is connected to the channel accumulator signal, and the multiplexer is connected to the sign bit comparator and the max pooling logic module signal respectively; the sign bit comparator is used to detect the sign bit of the operation result output by the channel accumulator, determine whether the operation result is positive or negative, and output a judgment signal to the multiplexer; the multiplexer performs a gating operation according to the judgment signal: when the operation result is positive, it gating and outputs the operation result; when the operation result is negative, it gating the zero signal, forcing the operation result to zero and outputting it.

[0109] (d') MaxPool Logic: This module is used for data filtering and compression. It utilizes registers to compare the size of adjacent data in real time during data flow, retaining only the most significant feature (maximum value) within a local region, thereby reducing the amount of subsequent data processing. Based on the pool_phase control bit, Phase 0 only temporarily stores the data in the Temp Buffer; Phase 1 first performs a vertical comparison (V_{max} = Max(Curr, Temp)), and then performs a horizontal comparison using the internal register pool_reg. Only a small amount of comparator resources are needed to complete the maximum value extraction and dimensionality reduction of a 2x2 region during data flow.

[0110] The max pooling logic module includes a horizontal comparator and a vertical comparator. The horizontal comparator is connected to the pooling temporary storage area of ​​the local circular buffer, and the vertical comparator is connected to both the horizontal comparator and the result storage area of ​​the local circular buffer. The horizontal comparator performs pairwise comparisons on each row of data within a preset pooling window, extracts the maximum value of each row, and sends it to the vertical comparator. The vertical comparator compares the maximum values ​​of each row extracted by the horizontal comparator again, and finally extracts the maximum value of the data within the entire pooling window, which is then output as the feature value of the pooling window to the result storage area of ​​the local circular buffer.

[0111] Furthermore, the pooling process of the max pooling logic module is as follows:

[0112] When a 2x2 max pooling window (MaxPool(2x2)) is enabled, the FSM selects the working phase based on the pool_phase parameter:

[0113] Phase 0 (pool_phase=0, even rows): Temporarily store the convolution output in Temp Buffer (index 64-79);

[0114] Phase 1 (pool_phase=1, odd-numbered rows): The current row output is compared vertically with the corresponding position in the Temp Buffer, and then horizontally between adjacent columns. The final result is written to the first half of the Output Region (index 48-55).

[0115] The hybrid-mode state machine is the microsecond-level scheduling hub within the coprocessor responsible for workflow control. The RS instruction, after parsing, provides the hybrid-mode state machine, which dynamically reuses and reconstructs data paths by receiving instruction configuration parameters (such as fc_mode, acc_en, etc.), achieving "one set of hardware, dual functionality." Specifically, the coprocessor selects the operating mode based on the fc_mode parameter in the RS.CALC instruction: when fc_mode=0, it enters convolution mode (S_CALC, convolutional streaming state); when fc_mode=1, it enters fully connected mode (S_CALC_FC, fully connected iterative state).

[0116] The UML diagram of a hybrid mode state machine is as follows: Figure 3 As shown, its finite state machine transition design is as follows:

[0117] (a'') IDLE (Idle State): The initial system state. In this state, the coprocessor sends a "Ready" signal to the instruction dispatcher. A state transition is triggered when a valid RS.CALC instruction is received and written to the register via the instruction configuration interface. The mixed-mode state machine selects the mode based on the fc_mode flag in the instruction: if fc_mode=0, it jumps to the S_CALC state; if fc_mode=1, it jumps to the S_CALC_FC state.

[0118] (b'') S_CALC (Convolutional Streaming State): In this state, the coprocessor sends a "Busy" signal to the dispatcher and blocks new external instructions. The built-in cycle counter calc_cnt (counting range 0 to 15) is started. The hardware is configured for combinational logic through-flow mode, completing one dot product, accumulation, activation, and pooling operation for a 3x3 sliding window each clock cycle. When calc_cnt reaches 15 (or the maximum value set for a row), it means the current image row has been calculated. The combinational logic of the hybrid mode state machine is reset, triggering a completion interrupt or signal, and automatically jumping back to the IDLE state.

[0119] (c'') S_CALC_FC (Fully Connected Iterative State): In this state, the coprocessor also sends a "Busy" signal. The hybrid-mode state machine starts its built-in counter idx (counting range 0 to 15). Unlike the convolutional streaming state, in this state, the hybrid-mode state machine cuts off the subsequent ReLU and pooling pipeline stages and dynamically refactors the combinational logic of the nine multipliers into a serial iterative mode that activates only one multiplier (such as multiplier 0). Each cycle, idx is used as an address pointer to read a pair of vector elements from the buffer for multiplication and addition. When idx reaches 15, the result is latched into the result area, the hybrid-mode state machine ends the computation cycle, releases the multiplier lock, triggers the instruction completion interrupt, and automatically jumps back to the "IDLE" state.

[0120] Figure 3 In the process, after system reset (Reset), the hybrid mode state machine is in the IDLE state and outputs Ready=1 to the instruction dispatcher. When the coprocessor receives the RS.CALC instruction at the decoding interface and fc_mode==0, it enters S_CALC (convolutional streaming state) and sends a Busy=1 signal to the instruction dispatcher. In S_CALC mode, the internal counter calc_cnt is initialized to 0. On the rising edge of the clock, the pipeline runs in parallel according to the preset convolution window 3. 3. Read data, perform addition tree dot product operation, trigger channel accumulation, activation function and pooling logic operation according to the configured instruction parameters, and after the row data (e.g. row width 15) is completed, the coprocessor sends a "combinational logic reset, trigger instruction completion interrupt" message to the instruction dispatcher.

[0121] In step (33), under the convolution mode, the specific implementation of each operation is as follows:

[0122] The convolution operation involves multiplying the data to be processed in the local circular buffer and the corresponding weight parameters in the shadow weight register using a preset convolution kernel, and then summing the results of each multiplication operation to generate an initial convolution response value.

[0123] The accumulation operation adds the initial convolution response value generated this time to the previous convolution response value stored in the result storage area of ​​the local circular buffer, updates the response value, and stores it back in the result storage area;

[0124] The activation operation involves checking the sign bit of the accumulated response value. If a negative response value is detected, the response value is forcibly set to zero, and a non-negative response value is retained as the activation result. Otherwise, the response value is retained as the activation result.

[0125] Pooling operation extracts the maximum value among all data within a preset pooling window, using it as the feature value of the current pooling window data, and stores it in the result storage area.

[0126] Furthermore, the workflow of the convolution mode (i.e., the S_CALC convolutional streaming state) in step (33) is as follows:

[0127] (331') The state machine reads aligned 3x3 image window data directly and in parallel from the row storage area (Input Row 0 / 1 / 2) of the local circular buffer according to the built-in counter calc_cnt; 9 multipliers perform multiplication operations of weights and pixel data in parallel, and the subsequent addition tree sums all the product results to generate the initial convolution response value of the current channel:

[0128] (332') If the acc_en enable bit in the instruction is valid, the hardware circuit automatically establishes a readback path, reads the previous calculation result (partial sum) from the target address of the LocalBuffer, and adds it to the current convolution response value to realize hardware-level fusion of multi-channel features;

[0129] (333') If the enable bit relu_en is valid in the instruction, the sign bit of the data is checked, and the negative result is forced to zero to realize the nonlinear transformation;

[0130] (334') If pooling is enabled, the step-by-step operation is performed according to the phase bit (pool_phase). In Phase 0, the result is temporarily stored in the Temp Buffer. In Phase 1, the pipeline maximum value filtering of "vertical comparison + horizontal comparison" is performed, and the filtered feature value is finally written to the result area. If pooling is not enabled, the data is directly written to the Local Buffer output area.

[0131] (335') The built-in counter calc_cnt corresponds to the horizontal width of the current processing row (in this embodiment, it is fixed at 16 clock cycles). It increments from 0 to 15, and after completing the horizontal traversal calculation of the current row of data pixel by pixel, the state machine automatically returns to the idle state, waiting for the CPU to issue instructions for the next row of data.

[0132] Figure 3 In the process, when the coprocessor receives the RS.CALC instruction at the decoding interface and fc_mode=1, it enters S_CALC_FC (fully connected iterative state) and sends a Busy=1 signal to the instruction dispatcher. In S_CALC_FC mode, pipeline cascading is cut off, multiplier 0 is activated, the FC dedicated accumulator fc_acc=0, index idx=0, and serial multiplication and addition are performed on the rising edge of the clock: fc_acc += (Local_Buffer[idx]). Weight_Reg[idx]), idx++, complete the serial multiplication and addition operation of the vector length number (e.g., 15), write the result back to the local circular buffer, Local_Buffer

[48] =fc_acc, the coprocessor reports "Release multiplier lock to trigger instruction completion interrupt" to the instruction dispatcher.

[0133] In the fully connected mode of step (33), the implementation steps of the single-core serial multiplexing are as follows:

[0134] (331) Activate one of the idle coprocessors as a busy coprocessor, and activate one of the multiplication functions of the busy coprocessor (such as the 0th multiplier of the target coprocessor) to specifically perform vector dot product operations in fully connected mode;

[0135] (332) Cut off the functions of accumulation, activation and pooling operations in the busy coprocessor, and retain only the convolution operation function. Unfold the convolution operation result and load it into the result storage area provided by the busy coprocessor. At the same time, load the pre-trained weight parameters corresponding to the convolution operation result into the weight parameter storage area of ​​the local circular buffer of the busy coprocessor.

[0136] (333) The convolution operation unrolling result and the corresponding pre-trained weight parameters loaded in step (332) will reside in the corresponding data storage area throughout all vector dot product operations in the subsequent fully connected mode, and will not be loaded again. The busy coprocessor performs vector dot product operations on the loaded convolution operation unrolling result and the corresponding pre-trained weight parameters. After completing the operation for one image category, it switches to the next image category and repeats the above vector dot product operation process until all image categories have been operated.

[0137] Further, the workflow of the fully connected mode (fully connected iterative state S_CALC_FC) in step (33) is as follows: the hybrid modular state machine FSM reconfigures the data path, routing Local_Buffer[idx] and Weight_Reg[idx] only to multiplier number 0; clears the dedicated accumulator fc_acc for FC; the built-in counter idx counts from 0 to 15, performing a multiply-accumulate operation once per cycle; after the loop ends, the value of fc_acc is written back to Local Buffer index 48.

[0138] In this invention, the coprocessor's instruction parsing, mixed-mode state machine, mode selection, FC dedicated accumulator, and arithmetic logic unit form an operator fusion pipeline, such as... Figure 1 As shown. This invention achieves hardware iteration acceleration of the fully connected layer through a hybrid mode state machine. It achieves the reuse of the computation path with minimal control logic overhead, avoiding the performance precipitate caused by traditional lightweight accelerators that cannot handle discrete vector calculations and thus roll back the task to the CPU. It effectively solves the Amdahl bottleneck and makes the overall inference pipeline more balanced.

[0139] The data flow during the execution of this invention is as follows: Figure 4 As shown, the workflow of this invention is described using the MNIST handwritten digit recognition network (LeNet-5 variant) as an example:

[0140] Phase 1: System Initialization

[0141] The main processor (CPU core) uses the standard bus to access the RS instruction to load the pre-trained neural network weight parameters and the raw input image data to be inferred from the external non-volatile storage medium (such as Flash or SD card) into the designated address area of ​​DRAM, completing the initial deployment of the data. At this time, the coprocessor is in an idle standby state.

[0142] Second stage: Feature extraction layer (convolution and pooling)

[0143] (1) Execute the first layer (16x16->8x8):

[0144] For each coprocessor core i (i=0-7):

[0145] The default pooling window is 2x2, and each coprocessor is responsible for calculating 1 output row (corresponding to the processing of 2 input rows).

[0146] For each coprocessor i (i=0-7):

[0147] Execute the RS.DMA instruction to load the second channel of the single-channel input image. i-1, 2 i, 2 The image data of line i+1 is loaded into the 0-15, 16-31, and 32-47 regions of the Local Buffer;

[0148] The RS.DMA instruction is executed to load the preset 3x3 convolution kernel and the weight parameters corresponding to the convolution kernel data into the 80 starting area of ​​the LocalBuffer (as intermediate storage).

[0149] The RS.WLOAD instruction is executed to load the weights into the shadow weight register;

[0150] Execute the RS.CALC command (acc_en=0, relu_en=0, pool_phase=0) to calculate even-numbered rows (Row 2). i) Convolution output and write to Temp Buffer. Since it's a single-channel input (MNIST), the data is directly overwritten, and no non-linear activation is performed in the first stage.

[0151] Execute the RS.DMA instruction to load the next line (Row 2). i+2) covers the oldest row;

[0152] Execute the RS.CALC instruction (acc_en=0, relu_en=1, pool_phase=1) to compute the odd-row convolution (Row2). i+1) and merge with Temp Buffer, then activate relu_en=1 (ReLU=Max(0,max)) and write the result to the Output Region.

[0153] Execute the RS.DMA instruction (is_write=1) to write the Output Region back to DRAM.

[0154] If it is a multi-channel input (such as RGB input), a loop needs to be added before RS.CALC above:

[0155] For each channel ch (ch=0 to C-1), C is the number of channels:

[0156] If ch < C-1 (middle channel):

[0157] Executing RS.CALC (acc_en=1, relu_en=0) -> accumulates but does not activate, resulting in a partial sum;

[0158] If ch == C-1 (last channel):

[0159] Execute RS.CALC (acc_en=1, relu_en=1) -> accumulate and activate, generating the final FeatureMap.

[0160] (2) Execute the second layer (8x8->4x4):

[0161] Since the output feature map size is reduced to 4x4, the system adopts a "logical folding" strategy, activating only the first four processing coprocessors (coprocessor 0 to coprocessor 3) to work in parallel, while the remaining coprocessors automatically enter a low-power standby state (idle state). Each coprocessor is responsible for calculating one output row.

[0162] For each activated coprocessor i (i=0-3):

[0163] The RS.DMA instruction is executed to load the weights of the second-layer convolutional kernels corresponding to the six input channels into the local buffer weight reservation storage area of ​​each coprocessor;

[0164] For each input channel ch (ch = 0 to 5, corresponding to the 6 feature map channels of the previous layer's output):

[0165] Execute the RS.DMA instruction to load the second feature from the current channel of the 8x8 input feature map. i-1, 2 i, 2 i+1 rows of data;

[0166] Determine the accumulation and activation conditions:

[0167] If ch == 0 (first channel): acc_en = 0 (overwrite old value), relu_en = 0 (intermediate results not activated).

[0168] If 0 < ch < 5 (intermediate channel): acc_en = 1 (accumulate old value), relu_en = 0 (intermediate result not activated)

[0169] If ch == 5 (last channel): acc_en = 1 (accumulate old value), relu_en = 1 (activate final result).

[0170] The RS.CALC instruction (applying the acc_en / relu_en parameters set in the current loop, and pooling_en=0 when ch<5, pooling_en=1 when ch=5; specifying pool_phase=0) is executed to perform convolution calculations on even-numbered rows within the window. At this point, if acc_en=1, channel accumulation is performed; if pooling_en=0, the semi-finished data is temporarily stored; if ch=5, the pooling operation is prepared, and the result is temporarily stored in the Temp Buffer.

[0171] Execute the RS.DMA instruction to load the next line of data;

[0172] The RS.CALC instruction (applying the acc_en / relu_en parameters set in the current loop, and pooling_en=0 when ch<5, pooling_en=1 when ch=5; specifying pool_phase=1) is executed to perform convolution calculations on odd-numbered rows. Convolution accumulation is only performed in the first / middle channels; in the last channel (ch=5), the hardware reads the even-numbered rows from the Temp Buffer and performs 2x2 Max Pooling with the current odd-numbered rows, finally applying ReLU activation and writing the final feature values ​​to the Output Region.

[0173] Execute the RS.DMA instruction (is_write=1) to write the calculated i-th row of output features back to the second layer of DRAM result area.

[0174] The third stage: Fully connected classification layer (4x4->10) adopts a single-core serial multiplexing strategy and uses coprocessor 0 to sequentially calculate the scores of 10 categories.

[0175] For each category i (i=0-9):

[0176] (1) Input loading (outside the loop):

[0177] The RS.DMA instruction loads the flattened 16 input feature values ​​(4x4 Feature Map) into the Input Row 0 area (index 0-15) of the Local Buffer of Coprocessor 0 in one go. Subsequent calculations can reuse these features without repeated loading, thus minimizing I / O overhead.

[0178] (2) Weight traversal calculation (within the loop):

[0179] For each category i (i=0-9):

[0180] Execute the RS.DMA instruction to read the 16 weight parameters corresponding to the current category from DRAM and load them into the weight reservation area of ​​LocalBuffer (starting from index 80).

[0181] The RS.WLOAD instruction is executed to load the weight data from the Local Buffer into the internal shadow weight register in parallel.

[0182] Execute the RS.CALC instruction (fc_mode=1) to start the fully connected computation mode. The hardware uses a counter to automatically perform a vector dot product on Input Row 0 and the shadow weight, and writes the final scalar score (Logit) back to the first address (index 48) of the Output Region.

[0183] Execute the RS.DMA instruction (is_write=1) to write the single score value at the Output Region back to the corresponding position in the result array in DRAM.

[0184] Figure 4 In the text, "16x10" indicates the relationship between the feature input of the fully connected layer and the classification category: that is, the 4x4 (16 after flattening) feature values ​​output by the network are combined with the weights corresponding to 10 different image categories for calculation; "10 confidence scores" represent the scalar scores that the fully connected layer outputs after completing the vector dot product iteration operation, which correspond to 10 image categories (such as handwritten digits 0-9). The level of this score reflects the probability that the image belongs to a specific category.

[0185] The memory arbitrator of this invention adopts a fixed priority arbitration strategy. The coprocessor initiates a request with mem_req, mem_wen indicates the read / write direction, mem_ack indicates that the transfer handshake is complete, DMA is transferred in a burst form per word, and the internal counter dma_cnt automatically ends after decrementing to 0.

[0186] For neural network layers of different sizes, this invention achieves universal support through the following mechanisms: Support for images of arbitrary width: A strip mining strategy is adopted, and when the image width is greater than the number of coprocessors, the image is segmented along the width direction and processed in blocks; Support for variable convolution kernel size: For large convolution kernels such as 5x5 and 7x7, a software-based sliding window decomposition strategy is adopted to decompose the large convolution into multiple 3x3 operations; Support for multi-channel convolution: The local buffer in-situ accumulation mechanism is implemented through the acc_en control bit and channel accumulator, supporting multi-channel partial sum accumulation.

Claims

1. A method for controlling a RISC-V based neural network co-processor, characterized in that, Includes the following steps: (1) Define access instructions, compound operation instructions and loading instructions based on RISC-V architecture; load external pre-trained weight parameters and image data to be inferred; (2) Allocate a local circular buffer and a shadow weight storage area; the local buffer space is used to cache the data to be processed and the calculation results, and the shadow weight storage area is used to cache the weight parameters to be called; (3) Combining the pre-trained weight parameters, the image data to be inferred is processed in parallel on a row-by-row basis. For each row of data in the image to be inferred, the processing steps are as follows: (31) Execute the access instruction, preset the convolution window, and load the data of the convolution window size and the corresponding pre-trained weight parameters into the local circular buffer; (32) Execute the loading instruction to load the pre-trained weight parameters cached in the local circular buffer into the shadow weight storage area for subsequent calculations; (33) Execute the compound operation instruction and perform the corresponding operation according to the operation mode: For the convolution mode, perform one or more of the operations of convolution, accumulation, activation and pooling according to the parameters provided by the compound operation instruction; For the fully connected mode, for each image category, reload the pre-trained weight parameters corresponding to the category into the weight parameter storage area of ​​the local circular buffer, and then perform vector dot product operation on each image category in a single-core serial multiplexing manner.

2. The RISC-V-based neural network coprocessor control method according to claim 1, characterized in that: In step (3), a number of coprocessors greater than or equal to the total number of rows of the image data to be inferred are configured, and processors in an idle state are scheduled to execute steps (31), (32), and (33) in parallel on a row-by-row basis to achieve parallel processing of multiple rows of data.

3. The RISC-V-based neural network coprocessor control method according to claim 1, characterized in that: In step (31), the local circular buffer is configured with the following storage areas: The data storage area is used to store the data to be processed corresponding to the current convolution window; The results storage area is used to store the operation results and intermediate iteration results in convolution mode and fully connected mode; The weight parameter storage area is used to store the pre-trained weight parameters that correspond one-to-one with each image category.

4. The RISC-V-based neural network coprocessor control method according to claim 3, characterized in that: In step (33), under the convolution mode, the specific implementation of each operation is as follows: The convolution operation involves multiplying the data to be processed in the local circular buffer and the corresponding weight parameters in the shadow weight register using a preset convolution kernel, and then summing the results of each multiplication operation to generate an initial convolution response value. The accumulation operation adds the initial convolution response value generated this time to the previous convolution response value stored in the result storage area of ​​the local circular buffer, updates the response value, and stores it back in the result storage area; The activation operation involves checking the sign bit of the accumulated response value. If a negative response value is detected, the response value is forcibly set to zero, and a non-negative response value is retained as the activation result. Otherwise, the response value is retained as the activation result. Pooling operation extracts the maximum value among all data within a preset pooling window, using it as the feature value of the current pooling window data, and stores it in the result storage area.

5. The RISC-V-based neural network coprocessor control method according to claim 4, characterized in that: In the fully connected mode of step (33), the implementation steps of the single-core serial multiplexing are as follows: (331) Activate one of the idle coprocessors as a busy coprocessor, and activate one of the multiplication functions of the busy coprocessor to specifically perform vector dot product operations in fully connected mode; (332) Cut off the functions of accumulation, activation and pooling operations in the busy coprocessor, and retain only the convolution operation function. Unfold the convolution operation result and load it into the result storage area provided by the busy coprocessor. At the same time, load the pre-trained weight parameters corresponding to the convolution operation result into the weight parameter storage area of ​​the local circular buffer of the busy coprocessor. (333) The convolution operation unrolling result and the corresponding pre-trained weight parameters loaded in step (332) will reside in the corresponding data storage area throughout all vector dot product operations in the subsequent fully connected mode, and will not be loaded again. The busy coprocessor performs vector dot product operations on the loaded convolution operation unrolling result and the corresponding pre-trained weight parameters. After completing the operation for one image category, it switches to the next image category and repeats the above vector dot product operation process until all image categories have been operated.

6. A coprocessor architecture for executing the RISC-V-based neural network coprocessor control method according to any one of claims 1-5, characterized in that: It includes an instruction distributor and at least one coprocessor; the number of coprocessors is greater than or equal to the total number of rows of the image data to be inferred; the instruction distributor is electrically connected to each coprocessor and is used to schedule the working state of each coprocessor. When each coprocessor is idle, the image data to be inferred is processed in parallel on a line-by-line basis under the allocation and scheduling of the instruction allocator. The instruction distributor is equipped with an instruction sliding buffer unit, which is signal-connected to each coprocessor. The instruction sliding buffer unit is used to temporarily store instructions to be sent to the target coprocessor when the target coprocessor is busy, and to send the cached instructions to the coprocessor when the target coprocessor is idle. Each coprocessor interacts with the instruction distributor to receive instructions and perform the arithmetic processing described in steps (31), (32), and (33).

7. The coprocessor architecture according to claim 6, characterized in that: Each coprocessor includes a distributed memory unit, a shadow weight register group, an arithmetic logic unit, and a hybrid mode state machine, with signal interaction between the components; The distributed storage unit has a built-in local circular buffer, which interacts with the shadow weight register signal to store the row data to be processed, weight parameters, and calculation results. The shadow weight register group interacts with the arithmetic logic unit via signals. The shadow weight register group provides a shadow weight storage area for storing the pre-trained weight data required for the current operation of the arithmetic logic unit. The arithmetic logic unit is connected to the hybrid mode state machine signal. The arithmetic logic unit integrates various operation modules, including: a convolution kernel operator for performing convolution operations, a channel accumulator for performing multi-channel partial summation, an activation function module for correcting data signs, and a max pooling logic module for filtering and compressing data. The hybrid-mode state machine is used to switch the coprocessor's operation mode, including convolution mode and fully connected mode, and is used to output a mode selection signal to the arithmetic logic unit. The arithmetic logic unit performs the operation in the corresponding mode according to the mode selection signal.

8. The coprocessor architecture according to claim 7, characterized in that: The convolution kernel operator includes nine 32-bit multipliers and one adder tree; the nine 32-bit multipliers are all connected to the shadow weight register group and the data storage area of ​​the local circular buffer, and are also connected to the adder tree. Each multiplier operates in parallel, receiving weight data from the shadow weight register and data of the convolution window size from the data storage area of ​​the local circular buffer, and synchronously performing element-wise dot product operations; the addition tree sums the results of each dot product operation and outputs the initial response value of the convolution operation.

9. The coprocessor architecture according to claim 7, characterized in that: The activation function module includes a sign bit comparator and a multiplexer; the sign bit comparator is connected to the channel accumulator signal, and the multiplexer is connected to the sign bit comparator and the max pooling logic module signal respectively; The sign bit comparator is used to detect the sign bit of the operation result output by the channel accumulator, determine whether the operation result is positive or negative, and output a judgment signal to the multiplexer. The multiplexer performs a gating operation based on the judgment signal: when the operation result is positive, it selects and outputs the operation result; when the operation result is negative, it selects the zero signal, forces the operation result to zero and outputs it.

10. The coprocessor architecture according to claim 7, characterized in that: The max pooling logic module includes a horizontal comparator and a vertical comparator; the horizontal comparator is connected to the pooling temporary storage area of ​​the local circular buffer, and the vertical comparator is connected to the horizontal comparator and the result storage area of ​​the local circular buffer, respectively. The horizontal comparator performs pairwise comparisons on each row of data within the preset pooling window, extracts the maximum value in each row, and sends it to the vertical comparator. The vertical comparator then compares the maximum values ​​extracted by the horizontal comparator on each row again, and finally extracts the maximum value of the data within the entire pooling window, which is then output as the feature value of the pooling window to the result storage area of ​​the local circular buffer.

Images