Binary network accelerator with floating point coprocessor and programmable control method

Abstract

According to the binary network accelerator with the floating-point coprocessor and the programmable control method, the floating-point coprocessor is integrated on a chip, so that the supporting capacity for extra floating-point operation is remarkably improved. The accelerator adopts a three-stage pipeline architecture, floating point operation before convolution, binary convolution and floating point operation after convolution are respectively processed, and all stages are coordinated by a special control module, so that efficient parallel processing is realized. The floating point coprocessor has time and space parallelism, is composed of a plurality of vector group units, and supports general floating point operation. Accelerator programmable control can prevent circuit layout and wiring from being carried out again when working loads are switched. The accelerator is divided and executes two types of sub-graphs in a pipeline mode, various network topologies and residual connection are compatible, the limitation that a traditional accelerator is fixed in structure, poor in flexibility and the like is overcome, and the method can be widely applied to high-precision visual tasks.

Description

Technical Field

[0001] This invention relates to the technical field of simulation devices that use mathematical methods to calculate existing or anticipated conditions, and further relates to a binary network accelerator with a floating-point coprocessor and a programmable control method in the technical field of biological model-based computer systems. This invention can be used to implement programmability in accelerators that support high-precision binary network architectures, processing different types of floating-point operations while maintaining general floating-point capabilities and high parallel processing performance. Background Art

[0002] Binarized networks are an extreme form of lightweight convolutional networks. These networks partially or completely constrain weights or feature maps to between -1 and +1. This reduces network parameters and computational complexity, making them easier to deploy at the edge where resources and power are limited. The design of an accelerator for binarized networks must first ensure algorithm support—that is, whether the accelerator can support all operators in a specific binarized network to ensure functional completeness. Furthermore, given the potential need to switch between different network models at the edge, the accelerator should also provide a certain degree of programmability to avoid redesigning RTL code and layout and routing when switching workflows. Existing binarized network accelerators have two issues: one is functionality, especially for high-precision binarized networks that contain different types of additional floating-point operators. Existing binarized network accelerators lack support for these operators, making them incapable of supporting high-precision binarized networks, which reduces the recognition accuracy of visual algorithms. The other is programmability: most existing binarized network accelerators only support specific network models, lacking flexibility and adaptability to diverse workloads.

[0003] Xi'an Jiaotong University's patent application, "An Accelerator Structure and Loop Unrolling Method for Binarized Neural Networks" (Application Number: 202010631640.8, Authorization Announcement Number: CN 111797977 B), discloses a binary network accelerator with n-bit feature maps and 1-bit weights. The accelerator's computing units are divided into an array of convolutional computing units and an array of addition trees / accumulation units. The storage system is fully on-chip SRAM storage, including weight memory and feature map memory. The accelerator uses a pipeline architecture, with the pipeline starting from the weight and feature map SRAM, passing through the convolutional computing units and addition trees / accumulation units, and ending at the feature map SRAM. Although the accelerator provides high computational throughput and high scalability. However, the accelerator still has shortcomings. Since the accelerator's dedicated convolution module can only perform convolution operations between 1-bit weights and n-bit feature maps, it cannot handle floating-point operations in high-precision binary networks. In addition, the accelerator does not provide a dedicated module for processing additional floating-point operations, so it does not have the ability to process additional floating-point operations, resulting in low processing accuracy of convolutional networks.

[0004] Beijing Zhongke Shengxin Technology Co., Ltd. discloses a convolutional neural network accelerator architecture with binarized weights and activations in its patent application, "A Convolutional Neural Network Accelerator Architecture with Binarized Weights and Activations" (Application No. 201911077710.3, Authorization Announcement No. CN 111008691 B). This accelerator implements three convolutional layers (corresponding to 2, 6, and 10 operators) in hardware. The on-chip memory utilizes a dual-buffer architecture (3 / 4 cache pairs and 7 / 8 cache pairs). The advantages of this accelerator include: convolution weights and feature maps are all represented using 1-bit, and multiplication operations can be replaced with bitwise XOR, significantly reducing system hardware resource overhead. The system also utilizes a pipelined architecture to improve throughput. However, the accelerator still has a shortcoming: the three-stage pipeline of the accelerator is fixed, and each stage of the pipeline can only support one layer of convolution operation of a specific shape. Therefore, it can only support network topologies with three layers of convolution, but cannot support network topologies with more or less than three layers, let alone network topologies with branches or residual connections. The application of the accelerator is limited to simple binary networks with three-layer convolution structures.

[0005] In its patent application, "Hardware Accelerator for Binarized Convolutional Neural Networks and Data Processing Method Thereof" (Application No. 201911055498.0, Authorization Announcement No. CN 110780923 B), Hefei University of Technology discloses a binarized network accelerator architecture with a certain degree of versatility and support for some floating-point precision operations. This accelerator's support for floating-point precision operations is primarily reflected in the batch normalization operator. The accelerator combines the batch normalization and binarization operations immediately following the binarized convolution operation into an integer-precision comparison operation. The accelerator also employs a pipelined architecture, fetching feature maps and weight data from external memory or DDR memory. The data flows through a data preparation module, a convolution module, a batch normalization and binarization module, and a max pooling module before being output to external memory or DDR memory. The accelerator also leverages existing binarized convolution hardware to support the first convolutional layer and the final fully connected layer of the convolutional network. However, the accelerator still has a shortcoming: the batch normalization operation and the binarization operation are fused together, which makes the accelerator unable to support the network topology structure where the batch normalization and binarization operations are not adjacent to each other. The application of the accelerator is limited to the network structure where the batch normalization and binarization operations are adjacent to each other. Summary of the Invention

[0006] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a binary network accelerator with a floating-point coprocessor and a programmable control method to solve the problems in the prior art such as the lack of support for binary network floating-point operators, the accelerator's failure to support arbitrary network topologies, and poor programmability.

[0007] The technical approach to achieving the objectives of the present invention is to integrate an on-chip floating-point coprocessor, adding an additional vector floating-point coprocessor to a traditional binary network accelerator. To implement floating-point processing, the accelerator is also equipped with floating-point input caches (X-cache and Y-cache) and a floating-point output cache (FO). The floating-point coprocessor exhibits a certain degree of temporal parallelism and configurable spatial parallelism. Specifically, the floating-point coprocessor is composed of multiple vector group units, each of which employs a nine-stage computational pipeline. This temporal and spatial parallelism enables high throughput for the floating-point coprocessor. The accelerator includes a control subsystem comprising three control modules, one for controlling the binarization convolution operation, the floating-point operation before the binarization convolution operation, and the other for controlling the floating-point operation after the binarization convolution operation. The top layer of the control subsystem is a main controller (ctrl-m). The control logic of this main controller reads instructions one by one from its internal instruction RAM and distributes them to the three sub-control modules, initiating and coordinating the execution of the entire accelerator. The accelerator adopts a pipeline architecture from the top-level design, which is divided into three pipeline stages: pre-convolution floating-point operation stage flow-x, binarization convolution stage flow-b, and post-convolution floating-point operation stage flow-y, which are coordinated and controlled by the three sub-control modules mentioned above. The accelerator can execute a specific subgraph of the binarization convolution network in a pipeline at a time. The process of pipeline processing multiple subgraphs achieves the purpose of accelerating the calculation of the entire binarization convolution network. In order to realize the programmable control of the accelerator, the present invention proposes a programmable control method that can effectively divide the binarization convolution network into subgraphs and generate and distribute instructions. This method includes the definition of two types of subgraphs: one is a Class A subgraph consisting of pre-convolution floating-point operations, binarization convolution operations, and post-convolution floating-point operations, and the other is a Class B subgraph consisting only of continuous floating-point operations. This method also includes a hardware mapping method for the subgraphs, an instruction generation method, and a value execution and instruction distribution method. This control subsystem, hardware pipeline design, and programmable control method for the binary network can achieve programmability of the accelerator and can be used to accelerate binary convolutional networks with any number of layers, arbitrary topology, and residual connections. This solves the problem that traditional binary network accelerators only support binary convolutional networks with a fixed number of layers and no residual connections.

[0008] To achieve the above objectives, the present invention provides a binarized neural network accelerator comprising a binarized convolution unit, a vector floating-point coprocessor, a multiplexed DMA unit, and a programmable control subsystem that sends control signals to all of the above units, thereby implementing programmable control and pipeline execution. The vector floating-point coprocessor comprises a plurality of parameterizable vector group units, each of which internally employs a 9-stage arithmetic pipeline structure. The input of the multiplexed DMA unit is connected to the X port of the data input cache of the floating-point vector coprocessor, and the output of the X port is connected to the row cache in the binarized convolution unit. The input of the DMA unit or the output of the convolution unit is multiplexed to the Y cache of the floating-point cache and then to the Y data port of the floating-point vector coprocessor, and the output of the Y port is connected to the floating-point output FO cache of the DMA unit.

[0009] Furthermore, the binarized convolution unit includes two input caches, namely a weight cache and a feature map cache, an exclusive-or array and an addition tree unit, and a BO output cache; the binarized convolution unit is pipelined with a single output row as the slice granularity; wherein, the bireader module is responsible for transferring the data in the row cache to the feature map cache, and each time K rows of data are accumulated in the row cache, the bireader will read these K rows of input data into the feature map cache; the exclusive-or and addition tree units perform convolution operations on K rows of data each time to obtain integer-precision convolution accumulation results.

[0010] The XNOR array and addition tree unit have parallelism in three dimensions: Pn, Pm and Pc, where Pn represents the parallelism along the depth dimension of the input feature map, Pm represents the parallelism along the depth dimension of the output feature map, and Pc represents the parallelism along the output width dimension of the output feature map. Under this parallel architecture, its maximum throughput is Pc*Pm*Pn*2*freq, and each cycle of the unit can input up to Pc*Pm*Pn feature map / weight pairs and output up to Pm*Pc data results. Wherein, freq is the clock frequency. The weight and feature map inputs are connected to a row of XNOR gates and then to the addition tree. The output of the addition tree is corrected by 2*SUM-N and sent to the accumulator.

[0011] Furthermore, the vector floating-point coprocessor contains two groups of control signal ports, ctrl_x_vfp and ctrl_y_vfp, which correspond to the control information of flow-x and flow-y respectively. Each group of control signals includes a vector group programming request control signal, a vector group programming control signal, a vector group write back request control signal, and a vector group write back control signal.

[0012] Each vector group unit in the vector floating-point coprocessor integrates 32 PEG units, which perform parallel calculations on 32 floating-point elements respectively; each PEG contains an operation pipeline structure, which includes a multiplication pipeline and an addition pipeline. Both pipelines are 9-stage pipeline structures, and each stage processes a corresponding specific stage of multiplication or addition operations to shorten the critical path delay.

[0013] The pipeline execution refers to the three sub-processing pipelines of flow-x, flow-b and flow-y, which process each subgraph in parallel in a pipeline manner; among them, flow-x corresponds to the additional floating-point operations before the binarization convolution, flow-b corresponds to the binarization convolution operation, and flow-y corresponds to the additional floating-point operations after the binarization convolution.

[0014] Furthermore, each vector group unit in the vector floating-point coprocessor is dynamically divided during runtime, with a portion of the vector groups being assigned to the flow-x pipeline and the remainder being assigned to the flow-y pipeline. The division is performed in such a way that, if both the flow-x and flow-y stages exist, flow-x and flow-y divide the vector group units according to computing resource requirements; if only flow-y exists, all vector groups are assigned to flow-y.

[0015] Furthermore, the programmable control subsystem consists of a ctrl-m main controller and three sub-controllers: ctrl-b, ctrl-x, and ctrl-y; ctrl-m contains an instruction RAM unit, and the control logic in ctrl-m reads instructions from the instruction RAM unit and distributes them to the three sub-controllers; among them, ctrl-b is responsible for the coordinated control of the binary convolution unit, and ctrl-x and ctrl-y are responsible for the pipeline execution of flow-x and flow-y, respectively.

[0016] The steps of the programmable control method of the binary network accelerator of the present invention are as follows:

[0017] Step 1: Subgraph partitioning and hardware mapping of the binarized network: For a given binarized convolutional network, divide it into a finite number of subgraphs of two types, A and B. Type A subgraphs consist of pre-convolution floating-point operations, convolution operations, and post-convolution floating-point operations, while type B subgraphs consist only of floating-point operations. Map the pre-convolution floating-point operations in type A subgraphs to the flow-x stage of the binarized network accelerator, map the post-convolution floating-point operations in type A subgraphs to the flow-y stage of the binarized network accelerator, and map the convolution operations in type A subgraphs to the flow-b stage of the binarized network accelerator. Map the floating-point operations in type B subgraphs to the flow-y stage of the binarized convolutional network accelerator.

[0018] Step 2: Generate a subgraph instruction sequence: For a Class A subgraph, generate a first instruction containing the convolution parameter field and the slice field as well as the length information of the subsequent instruction, generate the operand load instruction and all floating-point operation instructions corresponding to the floating-point operation before the convolution, generate the operand load instruction and all floating-point operation instructions corresponding to the floating-point operation after the convolution, and concatenate all the above instructions to form the instruction sequence corresponding to the Class A subgraph; For a Class A subgraph, generate a first instruction containing the length information of the subsequent instruction, generate the operand load instruction and all floating-point operation instructions corresponding to the convolution, and concatenate all the above instructions to form the instruction sequence corresponding to the Class B subgraph;

[0019] Step 3: Transmit instructions and data: Concatenate the instruction sequences corresponding to each subgraph generated in step 2 to obtain the instruction sequence to be transmitted, and transmit the instruction sequence to be transmitted to the instruction RAM unit in the ctrl-m unit of the programmable control subsystem through a PCIe link and an on-chip AXI interconnect system; transmit the input feature map, weights, and floating-point parameter data of the subgraph or convolutional network to the DDR memory connected to the binarization convolution accelerator through the PCIe link and the on-chip AXI interconnect system;

[0020] Step 4, instruction fetch execution of the accelerator: The control logic in the ctrl-m unit in the programmable control subsystem reads the first instruction from the instruction RAM unit of the unit and determines the subgraph type. If it is a type A subgraph, the fields required by the binarization convolution unit are distributed to the ctrl-b unit in the programmable control subsystem and step 5 is executed. Otherwise, step 6 is executed.

[0021] Step 5: The control logic in the ctrl-m unit in the programmable control subsystem sequentially reads the operand load instructions corresponding to the floating-point operation before the convolution following the first instruction, as well as all floating-point operation instructions, and distributes these instructions one by one to the ctrl-x unit in the programmable control subsystem. After starting ctrl-x, execute step 6.

[0022] Step 6: The control logic in the ctrl-m unit in the programmable control subsystem sequentially reads the operand load instructions corresponding to the post-convolution floating-point operation and all floating-point operation instructions, and distributes these instructions one by one to the ctrl-y unit in the programmable control subsystem to start ctrl-y.

[0023] Compared with the existing technology, the present invention has the following advantages:

[0024] First, the present invention proposes to integrate a floating-point coprocessor on a chip and adopt a three-stage pipeline architecture to process pre-convolution floating-point operations, binarized convolution, and post-convolution floating-point operations respectively, thereby enhancing the support of the binarized network accelerator for additional floating-point operations. This overcomes the problem of low processing accuracy caused by the existing binarized network accelerator only supporting binarized convolution operators but not additional floating-point operations, enabling the accelerator of the present invention to support high-precision convolution networks with additional floating-point operations, thereby bringing higher visual processing accuracy.

[0025] Second, the programmable control method of the present invention can divide the complex binary network topology into subgraphs with simple structures, and can effectively map the subgraphs to the three-stage pipeline of the accelerator. The floating-point coprocessor in the present invention is a general-purpose processor with scalability, so that it can support all floating-point operations. The entire accelerator is executed in an instruction-based control manner. This general-purpose floating-point coprocessor design, general-purpose processor design based on instruction control, and programmable control method overcome the defects of the traditional binary network using a fixed hierarchical pipeline structure, not supporting residual connections, batch normalization and binary operation fusion, resulting in the lack of programmability of the hardware accelerator and the inability to adapt to a variety of different visual algorithms. Therefore, compared with existing accelerators, the accelerator proposed by the present invention has stronger flexibility and programmability, can be applied to different binary network topologies and different network models, and avoids the redesign of RTL code and layout and wiring when switching workflows. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the architecture of the binarized neural network accelerator of the present invention;

[0027] Figure 2 Schematic diagram of the XNOR array and addition tree structure of the convolution operation unit of the present invention;

[0028] Figure 3 Schematic diagram of the state machine of the convolution operation unit of the present invention;

[0029] Figure 4 Schematic diagram of the internal structure of the vector group unit of the present invention;

[0030] Figure 5 The present invention is a flowchart of a programmable control method for a binary network accelerator. DETAILED DESCRIPTION

[0031] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0032] The architecture of the binary network accelerator of the embodiment of the present invention is as follows: Figure 1As shown. The accelerator includes a binarized convolution unit, a vector floating-point coprocessor, a multiplexed DMA unit, and a programmable control subsystem that controls all units through an AXI bus to achieve programmable control and pipeline execution. The vector floating-point coprocessor is composed of a plurality of parameterizable vector group units. The input of the multiplexed DMA unit is connected to the input X cache of the floating-point vector coprocessor, and the output of the X cache is connected to the row cache in the binarized convolution unit. The input of the DMA unit or the output of the convolution unit is connected to the floating-point cache Y cache in a multiplexed manner and then to the Y cache of the floating-point vector coprocessor. The output of the Y cache is connected to the floating-point output FO cache of the DMA unit.

[0033] Reference Figure 2 , the structure of the binarized convolution unit in the embodiment of the binarized neural network accelerator with Pc=2, Pm=Pn=32, BO output cache precision of 16-bit, and frequency freq of 250MHz is described.

[0034] In the binarized convolution unit, since Pc=2, the feature map cache is divided into two banks to accommodate the parallelism requirements of the XNOR array and the adder tree unit in the Pc dimension. Each bank has a data width of 32 bits to accommodate the parallelism requirements of the XNOR array and the adder tree unit in the Pn dimension. In the binarized convolution unit, since Pm=Pn=32, the data width of the weight cache is 32x32=1024 bits to accommodate the parallelism requirements of the XNOR array and the adder tree unit in the Pm and Pn dimensions. In the binarized convolution unit, since Pm=32 and Pc=2, and the BO cache fixed-point precision is 16 bits, the BO cache is divided into two banks, each with a data width of 32x16 bits to accommodate the parallelism requirements of the Pm and Pc dimensions of the adder tree unit and the accumulated output precision requirements. Under this parallel architecture, the maximum throughput of the XNOR array and the addition tree unit is Pc*Pm*Pn*2*freq=1024GOPs.

[0035] Reference Figure 3 , the operation process of a row of convolution in the convolution operation unit state of an embodiment of the present invention is described, in which Pc=2, Pm=Pn=32 in the binary convolution unit, the binary convolution parameters are the number of rows C=112, the input depth N=output depth M=64, the convolution kernel size K=3, and the output depth dimension slice tileM=32.

[0036] After the binary convolution unit is controlled to start, it enters the Read Feature Map Handshake state from the Idle state. For each input data slice, that is, a continuous K = 3 rows of input feature maps, the unit first waits for these 3 rows of data. After the Read Feature Map Handshake is completed, it enters the Read Weight Handshake state, where the unit waits for the weight data of size KxKxtileMxN to be ready. In the embodiment of the present invention, this is 3x3x32x64.

[0037] After the weight data handshake is complete, the unit enters the working state. In the working state, the unit will calculate the convolution result of the three rows of data and the corresponding weight slice and write the result to the output buffer BO. After the convolution calculation of one row is completed, the unit determines that the output feature map depth 32 has not yet reached the maximum depth, so it enters the waiting weight state and waits for the next 3x3x32x64 weight slice.

[0038] In the write handshake state, a row of convolution calculations is repeated. After a row of convolution calculations is completed, the unit determines that the output feature map depth of 64 has reached the maximum output depth, and thus enters the write result handshake state.

[0039] After the handshake is completed, it is determined whether the current input feature map is the last row of convolution. If so, the entire unit ends its work and enters the idle state. Otherwise, it enters the read feature map handshake state and waits for the next K=3 rows of input feature maps to be ready.

[0040] The following is combined with Figure 4 , further describes the internal structure of a vector group unit in the vector coprocessor.

[0041] The vector floating-point coprocessor in this embodiment of the present invention contains two control signal ports, ctrl_x_vfp and ctrl_y_vfp, corresponding to flow-x and flow-y control information, respectively. These two control signal ports contain the same number and type of fields, so the ctrl_x_vfp control signal port in the vector floating-point coprocessor is used as an example to illustrate its operating principle: the ctrl_x_vfp control signal includes three programming request control signals: the ctrl_x_vfp_req signal is used to initiate a new programming request control, the ctrl_x_vfp_sel signal, issued along with this control signal, is used to select a specific vector group, and the final signal, ctrl_x_vfp_rel, is used to terminate the programming request control.

[0042] The ctrl_x_vfp signal contains five vector group programming control signals. ctrl_x_vfp_peg_restart is used to restart a vector group. The ctrl_x_vfp_peg_pipes signal, issued along with this control signal, specifies the pipeline depth (maximum PE unit pipeline depth of 9). The ctrl_x_vfp_fifowr signal is a write-valid signal for writing data from the floating-point input buffer to the vector group unit's internal FIFO. The vfp_din_x signal accompanying this write signal indicates the actual data written. The ctrl_x_vfp_ma_i_wr signal is a write signal for floating-point arithmetic instructions. The ctrl_x_vfp_ma_i signal accompanying this signal indicates the actual floating-point arithmetic instruction written. The ctrl_x_vfp signal also includes two vector group writeback request control signals. The ctrl_x_vfp_wback_req signal is used to initiate a new writeback request control. The accompanying ctrl_x_vfp_wback_sel signal is used to select a specific vector group. The ctrl_x_vfp signal also includes three vector group writeback control signals. The ctrl_x_vfp_wback_valid signal indicates that the writeback of the selected vector group unit is valid. The accompanying vfp_dout_x signal indicates the actual data written back. The ctrl_x_vfp_wback_ready signal indicates that the selected vector group unit is ready for writeback. The ctrl_x_vfp_wback_last signal indicates the end of the data writeback.

[0043] The working process of vector group division and pipeline division of an accelerator with 2 vector groups and type A acceleration subgraph in a floating-point coprocessor unit is further explained.

[0044] Because the Class A subgraph contains both pre-convolution and post-convolution floating-point operations, both the flow-x and flow-y pipeline stages exist. These stages divide the vector group units based on computational resource requirements. Since there are only two vector groups and each vector group cannot be further divided, flow-x is allocated one vector group unit and flow-y is allocated one vector group unit to ensure on-demand computational resource allocation. Because the Class A subgraph contains both pre-convolution and post-convolution floating-point operations as well as binarized convolution operations, the pre-convolution floating-point operations in the Class A subgraph are mapped to the flow-x pipeline stage, the binarized convolution operations are mapped to the flow-b pipeline stage, and the post-convolution floating-point operations are mapped to the flow-y pipeline stage.

[0045] Reference Figure 5The following is a flowchart of a programmable control method for a binarized network accelerator, using a specific Class A subgraph as an example. In this embodiment, the binarized neural network still uses binarized convolution parameters: number of rows C = 112, input depth N = output depth M = 64, convolution kernel size K = 3, and output depth dimension tile M = 32.

[0046] Step 1: Subgraph partitioning and hardware mapping of the binary network.

[0047] Assume that the pre-convolution floating-point operations in the A-type subgraph include a bias operator (BIAS), and the post-convolution floating-point operations include, in sequence, a scaling factor operator (SCALE), a batch normalization operator (BN), a residual addition operator (RES), and a bias and parameterized rectifier operator (BPRELU). Map the pre-convolution floating-point operation (BIAS) in the A-type subgraph to the flow-x stage of the binarization network accelerator. Map the post-convolution floating-point operations (SCALE+BN+RES+BPRELU) to the flow-y stage of the binarization network accelerator. Map the convolution operations in the A-type subgraph to the flow-b stage of the binarization network accelerator.

[0048] Step 2: Generate subgraph instruction sequence.

[0049] For the above-mentioned Class A subgraph, a first instruction is generated that contains the convolution parameter field, the slicing field, and the subsequent instruction length information. In this embodiment, the convolution parameter and slicing fields are: number of rows C = 112, input depth N = output depth M = 64, convolution kernel size K = 3, and output depth dimension slice tileM = 32. The operand load instruction OP_DATA and the floating-point operation instruction OP_BIAS corresponding to the floating-point operation before convolution are generated. The operand load instruction corresponding to the floating-point operation before convolution and the total length of the floating-point operation instructions are obtained and backfilled into the subsequent instruction length field of the first instruction. The operand load instruction OP_DATA and all floating-point operation instructions OP_SCALE+OP_BN+OP_RES+OP_BPRELU corresponding to the floating-point operation after convolution are generated. The operand load instruction and the total length of the floating-point operation instructions corresponding to the floating-point operation after convolution are obtained and backfilled into the subsequent instruction length field of the first instruction. All the above instructions are concatenated to form the instruction sequence corresponding to the Class A subgraph.

[0050] Step 3: Transmit instructions and data.

[0051] Assuming that only the above subgraph itself is accelerated, the instruction sequence corresponding to the subgraph in step 2 is transferred to the instruction RAM unit of the ctrl-m unit in the programmable control subsystem via a PCIe link and an on-chip AXI interconnect system. Then, the subgraph input feature map, namely the feature map before the BIAS operator and the feature map of the residual connection, the binary convolution weights, and the floating-point parameter data, namely the BIAS operator offset, SCALE scaling factor, the mean and standard deviation of BN, and the offset and slope of BPRELU, are transferred to the DDR memory connected to the binary convolution accelerator via the PCIe link and the on-chip AXI interconnect system.

[0052] Step 4: The accelerator fetches and executes instructions.

[0053] The control logic in the ctrl-m unit in the programmable control subsystem reads the first instruction from its instruction RAM unit and determines that the subgraph type is a Class A subgraph. Therefore, the required fields for the binarized convolution unit are set: number of rows C = 112, input depth N = output depth M = 64, kernel size K = 3, and output depth dimension tileM = 32. This information is distributed to the ctrl-b unit in the programmable control subsystem, and then step 5 is executed.

[0054] In step 5, the control logic in the ctrl-m unit in the programmable control subsystem sequentially reads the operand load instruction OP_DATA and the floating-point operation instruction OP_BIAS corresponding to the floating-point operation before the convolution following the first instruction, and distributes these instructions to the ctrl-x unit in the programmable control subsystem. After starting ctrl-x, execute step 6.

[0055] Step 6: The control logic in the ctrl-m unit in the programmable control subsystem sequentially reads the operand load instruction OP_DATA and the floating-point operation instructions OP_SCALE, OP_BN, OP_RES, and OP_BPRELU corresponding to the floating-point operation after convolution, and distributes these instructions one by one to the ctrl-y unit in the programmable control subsystem to start ctrl-y.

Claims

1. A binary network accelerator with a floating-point coprocessor, characterized in that: The accelerator includes a binarization convolution unit, a vector floating-point coprocessor, a multiplexed DMA unit, and a programmable control subsystem that controls all units via an AXI bus to achieve programmable control and pipeline execution. The vector floating-point coprocessor is composed of multiple parameterizable vector group units. The input of the multiplexed DMA unit is connected to the input X cache of the floating-point vector coprocessor, and the output of the X cache is connected to the row cache in the binarization convolution unit. The input of the DMA unit or the output of the convolution unit is connected to the floating-point cache Y cache in a multiplexed manner, and then to the floating-point cache Y cache of the floating-point vector coprocessor. The output of the Y cache is connected to the floating-point output FO cache of the DMA unit.

2. The binary network accelerator according to claim 1, characterized in that: The binary convolution unit includes two input caches of weight cache and feature map cache, an exclusive-or array and an addition tree unit, and a fixed-point precision BO output cache; the binary convolution unit is pipelined with a single output line as a slice granularity; The bireader module is responsible for transferring the data in the row cache to the feature map cache. Each time K rows of data are accumulated in the row cache, the bireader will read these K rows of input data into the feature map cache. The XNOR and ADD tree units perform convolution operations on K rows of data each time to obtain integer-precision convolution accumulation results.

3. The binary network accelerator according to claim 2, characterized in that: The XNOR array and addition tree unit have parallelism in three dimensions: Pn, Pm and Pc, where Pn represents the parallelism along the depth dimension of the input feature map, Pm represents the parallelism along the depth dimension of the output feature map, and Pc represents the parallelism along the output width dimension of the output feature map. Under this parallel architecture, its maximum throughput is Pc*Pm*Pn*2*freq, and each cycle of the unit can input up to Pc*Pm*Pn feature map / weight pairs and output up to Pm*Pc data results. Wherein, freq is the clock frequency. The weight and feature map inputs are connected to a row of XNOR gates and then to the addition tree. The output of the addition tree is corrected by 2*SUM-N and sent to the accumulator.

4. The binary network accelerator according to claim 3, characterized in that: The vector floating-point coprocessor contains two groups of control signal ports, ctrl_x_vfp and ctrl_y_vfp, which correspond to the control information of flow-x and flow-y respectively. Each group of control signals includes a vector group programming request control signal, a vector group programming control signal, a vector group write back request control signal, and a vector group write back control signal.

5. The binary network accelerator according to claim 4, characterized in that: Each vector group unit in the vector floating-point coprocessor integrates 32 PE units, which perform parallel calculations on 32 floating-point elements respectively; each PE unit contains an operation pipeline structure, which includes a multiplication pipeline and an addition pipeline. Both pipelines are 9-stage pipeline structures, and each stage processes a corresponding specific stage of multiplication or addition operations to shorten the critical path delay.

6. The binary network accelerator according to claim 5, characterized in that: The pipeline execution refers to the three sub-processing pipelines of flow-x, flow-b and flow-y, which process each subgraph in parallel in a pipeline manner; among them, flow-x corresponds to the additional floating-point operations before the binarization convolution, flow-b corresponds to the binarization convolution operation, and flow-y corresponds to the additional floating-point operations after the binarization convolution.

7. The binary network accelerator according to claim 6, characterized in that: Each vector group unit in the vector floating-point coprocessor is dynamically divided during runtime, with some vector groups being assigned to the flow-x pipeline and the rest being assigned to the flow-y pipeline. The division method is as follows: if both flow-x and flow-y stages exist, flow-x and flow-y divide the vector group units according to computing resource requirements; if only flow-y exists, all vector groups are assigned to flow-y.

8. The binary network accelerator according to claim 7, characterized in that: The programmable control subsystem consists of a ctrl-m main controller and three sub-controllers: ctrl-b, ctrl-x, and ctrl-y. ctrl-m contains an instruction RAM unit, from which the control logic reads instructions and distributes them to the three sub-controllers. ctrl-b is responsible for the coordinated control of the binary convolution unit, while ctrl-x and ctrl-y are responsible for the pipeline execution of flow-x and flow-y, respectively.

9. The programmable control method for a binary network accelerator according to claims 1-8, characterized in that: The steps of this method are as follows: Step 1: Subgraph partitioning and hardware mapping of the binarized network: For a given binarized convolutional network, divide it into a finite number of subgraphs of two types, A and B. Type A subgraphs consist of pre-convolution floating-point operations, convolution operations, and post-convolution floating-point operations, while type B subgraphs consist only of floating-point operations. Map the pre-convolution floating-point operations in type A subgraphs to the flow-x stage of the binarized network accelerator, map the post-convolution floating-point operations in type A subgraphs to the flow-y stage of the binarized network accelerator, and map the convolution operations in type A subgraphs to the flow-b stage of the binarized network accelerator. Map the floating-point operations in type B subgraphs to the flow-y stage of the binarized convolutional network accelerator. Step 2: Generate a subgraph instruction sequence: For a Class A subgraph, generate a first instruction containing the convolution parameter field and the slice field as well as the length information of the subsequent instruction, generate the operand load instruction and all floating-point operation instructions corresponding to the floating-point operation before the convolution, generate the operand load instruction and all floating-point operation instructions corresponding to the floating-point operation after the convolution, and concatenate all the above instructions to form the instruction sequence corresponding to the Class A subgraph; For a Class B subgraph, generate a first instruction containing the length information of the subsequent instruction, generate the operand load instruction and all floating-point operation instructions corresponding to the convolution, and concatenate all the above instructions to form the instruction sequence corresponding to the Class B subgraph; Step 3: Transmit instructions and data: Concatenate the instruction sequences corresponding to each subgraph generated in step 2 to obtain the instruction sequence to be transmitted, and transmit the instruction sequence to be transmitted to the instruction RAM unit in the ctrl-m unit of the programmable control subsystem through a PCIe link and an on-chip AXI interconnect system; transmit the input feature map, weights, and floating-point parameter data of the subgraph or convolutional network to the DDR memory connected to the binarization convolution accelerator through the PCIe link and the on-chip AXI interconnect system; Step 4, instruction fetch execution of the accelerator: The control logic in the ctrl-m unit in the programmable control subsystem reads the first instruction from the instruction RAM unit of the unit and determines the subgraph type. If it is a type A subgraph, the fields required by the binarization convolution unit are distributed to the ctrl-b unit in the programmable control subsystem and step 5 is executed. Otherwise, step 6 is executed. Step 5: The control logic in the ctrl-m unit in the programmable control subsystem sequentially reads the operand load instructions corresponding to the floating-point operation before the convolution following the first instruction, as well as all floating-point operation instructions, and distributes these instructions one by one to the ctrl-x unit in the programmable control subsystem. After starting ctrl-x, execute step 6. Step 6: The control logic in the ctrl-m unit in the programmable control subsystem sequentially reads the operand load instructions corresponding to the post-convolution floating-point operation and all floating-point operation instructions, and distributes these instructions one by one to the ctrl-y unit in the programmable control subsystem to start ctrl-y.

Images