A memory based on in-memory CNN intermediate cache scheduling

Abstract

The present invention relates to a memory based on in-memory CNN intermediate cache scheduling, belonging to the field of multiplexed CNN scheduling operation technology, and solves the problems of high energy consumption and high latency caused by memory access and data transfer in the prior art. A memory based on in-memory CNN intermediate cache scheduling includes: a control logic unit for outputting a selection signal and an enable signal based on a clock signal and an external input valid flag signal; an operation logic unit for performing a convolution operation on elements and weights and outputting the corresponding convolution result to a selector; a selector for selecting to output the convolution result or the summation result based on the selection signal; an intermediate FIFO cache unit for selecting whether to write the data output by the selector into the intermediate FIFO cache unit and whether to output the stored data based on the enable signal; and a memory storage module for storing and calculating the summation result output by the first adder based on an external instruction signal and a storage address. This realizes the implementation of simplified CNN scheduling using a FIFO cache.

Description

Technical Field

[0001] The present invention relates to the field of multiplexed CNN scheduling operation technology, and in particular to a memory based on in-memory CNN intermediate cache scheduling. Background Art

[0002] In the era of big data, existing computing architectures face increased energy consumption and latency when faced with data-intensive applications due to large amounts of data transfer and bandwidth limitations. This results in vector parallelism on data being constrained by memory read and write bandwidth. To address the energy consumption and latency caused by memory access and data transfer, in-memory computing breaks through the traditional von Neumann bottleneck, achieving the integration of memory and logic units and becoming one of the primary technical approaches for intelligent computing. Academia has proposed SRAM-based arithmetic and logic circuits to implement arithmetic and logic operations in memory. However, existing SRAM-based arithmetic and logic circuits have a relatively simple computational model, limiting their performance when operating between external inputs and memory units. Current in-memory computing technologies face challenges such as hardware resource reuse.

[0003] A convolutional neural network is a deep neural network with a convolutional architecture. This structure reduces the amount of memory used by deep networks and the number of parameters, alleviating model overfitting. The convolution kernel is a crucial component of the convolutional layer. The kernel's essence is the feature extractor, whose primary function is to automatically extract deep information from the input signal. A convolutional layer consists of multiple feature planes, each of which is composed of multiple neurons. Each neuron is connected to a local region of the feature plane in the previous layer via a convolution kernel. The feature map of the previous layer is convolved with a learnable convolution kernel and then passed through an activation function to produce the output feature map. Each neuron is locally connected to neurons in the previous layer and shares weights. The output data is then deconvolved with the image using the shared weights and then a shared bias is added to each neuron. Without shared weights, each neuron would require a single convolution kernel and a single bias. Instead, each neuron is assigned the same convolution kernel and bias, significantly reducing the number of parameters and making it suitable for in-memory data reuse. Existing data transfer is between the logic operation unit and the storage unit. Large amounts of data transfer and bandwidth limitations lead to increased energy consumption and latency, making vector parallelism on data subject to the memory read and write bandwidth. Summary of the Invention

[0004] In view of the above analysis, an embodiment of the present invention aims to provide a memory based on in-memory CNN intermediate cache scheduling to solve the problems of energy consumption and delay caused by existing memory access and data transfer.

[0005] On the one hand, an embodiment of the present invention provides a memory based on in-memory CNN intermediate cache scheduling, including a control logic unit, a cache scheduling circuit unit and a memory storage module; the cache scheduling circuit unit includes an arithmetic logic unit, a first adder, a selector and an intermediate FIFO cache unit;

[0006] The control logic unit is used to output a selection signal and an enable signal according to a clock signal and an externally input valid flag signal;

[0007] The arithmetic logic unit is used to perform a convolution operation on the elements of the input array and the weights of the convolution kernel, and output the corresponding convolution result to the selector;

[0008] The first adder is configured to add the convolution result and the data output by the intermediate FIFO buffer unit based on an enable signal, and output the summed result to the selector or output the summed result to the memory storage module based on an enable negation signal;

[0009] The selector is configured to select and output the convolution result or the summation result based on the selection signal;

[0010] The intermediate FIFO buffer unit is configured to select whether to write the data output by the selector into the intermediate FIFO buffer unit and to select whether to output the stored data according to the enable signal;

[0011] The memory storage module is used to store and calculate the addition result output by the first adder according to the external instruction signal and the storage address.

[0012] Furthermore, the memory storage module includes a memory storage unit, a third adder and a temporary buffer area;

[0013] The memory storage unit includes n parts, each part storing a partial product which is a convolution result of a row of elements, and is used to store or read the partial product according to the external instruction signal and the storage address; wherein the convolution kernel size is n*n;

[0014] The third adder is used to add the partial products read from the memory storage unit to the corresponding convolution results to obtain a complete convolution result and store it in a temporary buffer area.

[0015] Furthermore, the number of the third adders and temporary buffer areas is consistent with the number of convolution kernels, both of which are Cout. Each third adder adds each part of the convolution result of the corresponding convolution kernel and stores it in the corresponding temporary buffer area.

[0016] Furthermore, the external command signal includes a memory read command signal memory-rd and a memory write command signal memory-wr;

[0017] When memory-rd=1 and memory-wr=0, the memory storage unit sequentially reads the n parts of data after the element is convolved with each convolution kernel and inputs them to the third adder; the third adder performs an addition operation and inputs them into the corresponding temporary buffer area;

[0018] When memory-rd=0 and memory-wr=1, the data in the temporary buffer is stored in memory;

[0019] When memory-rd=0 and memory-wr=0, the memory storage unit does not operate.

[0020] Furthermore, when the cache scheduling circuit performs an operation of an n*n convolution kernel size, the storage size of the intermediate FIFO cache unit is n.

[0021] Furthermore, the enable signal includes a FIFO read enable signal and a FIFO write enable signal;

[0022] The FIFO read enable signal is used to control the intermediate FIFO buffer unit to read data;

[0023] The FIFO write enable signal is used to control the intermediate FIFO buffer unit to write data or output data to the output end of the intermediate buffer scheduling circuit unit.

[0024] Furthermore, the selection signal is set to sel, the FIFO read enable signal is set to rd_en, and the FIFO write enable signal is set to wr_en. When used for the operation of the 3*3 convolution kernel and the valid flag signal is 1, the selection signal and enable signal output by the control logic unit include three stages:

[0025] Initial stage: first cycle: sel = 0, rd_en = 0, wr_en = 1;

[0026] Second cycle: sel=1, rd_en=1, wr_en=1;

[0027] The third cycle: sel = 0, rd_en = 0, wr_en = 1;

[0028] Cycle phase: fourth cycle: sel = 0, rd_en = 1, wr_en = 0;

[0029] Fifth cycle: sel=1, rd_en=1, wr_en=1;

[0030] Sixth cycle: sel=0, rd_en=0, wr_en=1;

[0031] …

[0032] End stage: last cycle: sel = 0, rd_en = 1, wr_en = 0;

[0033] Furthermore, when the data selection signal sel is 0, the control selector selects the convolution result to the intermediate FIFO buffer unit; when the data selection signal sel is 1, the control selector selects the sum of the convolution result and the frontmost data in the intermediate FIFO buffer unit to the intermediate FIFO buffer unit;

[0034] When the FIFO read enable signal rd_en is 1, the intermediate FIFO buffer unit performs a read operation, outputs the front data of the intermediate FIFO buffer unit to the first adder, and inputs the convolution result and the read data into the selector through the adder. When the FIFO read enable signal rd_en is 0, the intermediate FIFO buffer unit does not read;

[0035] When the FIFO write enable signal wr_en is 1, the intermediate FIFO cache unit performs a write operation to write the input data into the intermediate FIFO cache unit. When the FIFO write enable signal wr_en is 0, the intermediate FIFO cache unit is not written. At this time, the write enable is negated ~ wr_en = 1 is valid, indicating that the output result of the first adder to the output end of the cache scheduling circuit unit is valid, that is, the first adder outputs the result to the memory storage module.

[0036] Furthermore, the number of convolution kernels is 1, the number of third adders is 1, the number of temporary cache areas is 1, the memory storage unit has 3 partial products, and the memory storage unit sequentially stores the calculation results datah of each operation cycle output by the output end of the intermediate cache scheduling circuit unit; according to the memory read instruction signal, the memory storage unit reads the partial products of the corresponding elements in each part to the temporary cache area, and according to the memory write instruction signal, the memory storage unit writes the data in the temporary cache area back to the memory storage unit; wherein h is the ordinal number of the operation cycle; and each operation cycle includes the sum of the convolution results corresponding to the convolution of three groups of adjacent elements.

[0037] Furthermore, the number of the intermediate cache scheduling circuit units is 3, and the number of convolution kernels is Cout; the number of the third adder and the temporary cache area is Cout, and the input ends of the three intermediate cache scheduling circuit units respectively input the three rows of weights of the convolution kernel and the elements of the first row, and the three intermediate cache scheduling circuit units are completed in parallel, and the corresponding output ends sequentially output the calculation results datah1, datah2, and datah3 of the corresponding operation cycles; the calculation results datah1, datah2, and datah3 are respectively stored in the three partial products of the memory storage unit, and arranged in order according to the ordinal number of the convolution kernel. According to the memory read instruction signal, the memory storage unit reads the partial products of the corresponding elements in each part of each convolution kernel to the corresponding temporary cache area; according to the memory write instruction signal and the storage address, the memory storage unit writes the data in the temporary storage back to the memory storage unit; wherein, 1-3 is the ordinal number of the intermediate cache scheduling circuit unit.

[0038] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0039] 1. This application is based on the form of an in-memory vector processing unit, and part of the convolution operation process is performed in the memory, which reduces the bus between the storage unit and the external calculation, saves the bus bandwidth between the memory and the processor, and the bandwidth does not limit the internal parallel calculation. The configurability of the convolution vector can be fully utilized in the intermediate cache scheduling circuit device, and convolution support for parameters of arbitrary length can be achieved in multiple cycles.

[0040] 2. The scheduling logic of the intermediate cache is simplified. Through the first-in-first-out FIFO strategy and simple loop logic control, the control complexity of using SRAM or register file as the intermediate cache is reduced.

[0041] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0043] Figure 1 This is the overall architecture diagram of the intermediate cache scheduling memory for CNN in this application;

[0044] Figure 2 This is the intermediate cache circuit architecture diagram of this application;

[0045] Figure 3 Implement the model graph for the convolution acceleration module;

[0046] Figure 4 To cache the logic calculation graph;

[0047] Figure 5 It is an existing SIMD computing graph; DETAILED DESCRIPTION

[0048] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0049] A specific embodiment of the present invention discloses a memory based on in-memory CNN intermediate cache scheduling, such as Figure 1 and Figure 2 As shown,

[0050] It includes a control logic unit, a cache scheduling circuit unit and a memory storage module; the cache scheduling circuit unit includes an arithmetic logic unit, a first adder, a selector and an intermediate FIFO cache unit;

[0051] The control logic unit is used to output a selection signal and an enable signal according to a clock signal and an externally input valid flag signal;

[0052] The arithmetic logic unit is used to perform a convolution operation on the elements of the input array and the weights of the convolution kernel, and output the corresponding convolution result to the selector;

[0053] The first adder is configured to add the convolution result and the data output by the intermediate FIFO buffer unit based on an enable signal, and output the summed result to the selector or output the summed result to the memory storage module based on an enable negation signal;

[0054] The selector is configured to select and output the convolution result or the summation result based on the selection signal;

[0055] The intermediate FIFO buffer unit is configured to select whether to write the data output by the selector into the intermediate FIFO buffer unit and to select whether to output the stored data according to the enable signal;

[0056] The memory storage module is used to store and calculate the addition result output by the first adder according to the external instruction signal and the storage address.

[0057] Specifically, the intermediate FIFO buffer unit is a first-in-first-out queue, which simplifies the scheduling strategy.

[0058] Specifically, the memory based on the in-memory CNN intermediate cache scheduling, in which the intermediate cache scheduling circuit unit and the memory storage module are both in the memory in-memory unit; the input array and the convolution kernel are first calculated in the intermediate cache scheduling circuit unit after entering the memory, and the operation logic is divided into a start stage, a loop stage and an end stage; the data value output by the intermediate cache scheduling circuit unit is input into the memory storage unit, which stores it and takes the corresponding column of the convolution kernel row size and adds it and then stores it in a temporary cache area; that is, from the operation logic unit to the intermediate FIFO cache unit, and then enters the memory storage unit from the intermediate FIFO cache unit, there is no need to exit the entire memory in-memory unit, which reduces the bus bandwidth between the memory in-memory unit and the processor.

[0059] Specifically, the arithmetic logic unit includes cin multipliers and a second adder;

[0060] The i-th multiplier is used to perform a multiplication operation of Ai and Wi; wherein Ai is the i-th value in the element, and Wi is the i-th value in the weight; i∈[1,cin];

[0061] The second adder is used to obtain the sum of the output results of cin multipliers and output the sum result as the corresponding convolution result.

[0062] Specifically, such as Figure 4 As shown, the elements of the input arithmetic logic unit and the weights of the convolution kernel are determined according to the cache arithmetic logic; the valid flag signal is used to control whether the arithmetic logic unit writes the elements and the weights of the convolution kernel; when the valid flag signal is 0, it is not written, and when it is 1, it is written.

[0063] Specifically, the cache operation logic is to reduce the movement of elements A in the input array. An intermediate FIFO cache unit is selected to unilaterally change the convolution kernel weight W, thereby obtaining the convolution result of a row of elements in the input array and a row of weights of the convolution kernel.

[0064] Specifically, the rules for input elements and weights of cache operation logic are as follows:

[0065] Input values ​​for the first cycle: elements in the first row and first column and weights in the first row and first column;

[0066] Input values ​​for the second cycle: elements in the first row and second column and weights in the first row and second column;

[0067] Input values ​​for the third cycle: elements in the first row and second column and weights in the first row and first column;

[0068] Input values ​​for the fourth cycle: elements in the first row and third column and weights in the first row and third column;

[0069] Input values ​​for the fifth cycle: elements in the first row and third column and weights in the first row and second column;

[0070] Input values ​​for the sixth cycle: elements in the first row and third column and weights in the first row and first column;

[0071] …

[0072] Input value of the third-to-last period: elements in the first row and second-to-last column and weights in the first row and third column;

[0073] Input values ​​of the second-to-last period: elements in the first row and the second-to-last column and weights in the first row and the second column;

[0074] Input value of the last period: the element of the first row and the last column and the weight of the first row and the third column;

[0075] Starting from the seventh cycle, the element input value is shifted to the right by one every three cycles, and the weight is cycled in the same way as the input from the fourth cycle to the sixth cycle.

[0076] Specifically, the cache operation logic includes calling each column element in the input array in sequence by row, and obtaining several convolution results corresponding to the element based on the movement relationship between the element and the convolution kernel; wherein each element of the convolution kernel is a weight.

[0077] Specifically, the input and convolution result output of the one logic operation unit are expressed as:

[0078]

[0079] Among them, A is an element and W is a weight.

[0080] Furthermore, when the cache scheduling circuit performs an operation of an n*n convolution kernel size, the storage size of the intermediate FIFO cache unit is n.

[0081] Furthermore, the enable signal includes a FIFO read enable signal and a FIFO write enable signal;

[0082] The FIFO read enable signal is used to control the intermediate FIFO buffer unit to read data;

[0083] The FIFO write enable signal is used to control the intermediate FIFO buffer unit to write data or output data to the output end of the intermediate buffer scheduling circuit unit.

[0084] Furthermore, the selection signal is set to sel, the FIFO read enable signal is set to rd_en, and the FIFO write enable signal is set to wr_en. When used for the operation of the 3*3 convolution kernel and the valid flag signal is 1, the selection signal and enable signal output by the control logic unit include three stages:

[0085] Initial stage: first cycle: sel = 0, rd_en = 0, wr_en = 1;

[0086] Second cycle: sel=1, rd_en=1, wr_en=1;

[0087] The third cycle: sel = 0, rd_en = 0, wr_en = 1;

[0088] Cycle phase: fourth cycle: sel = 0, rd_en = 1, wr_en = 0;

[0089] Fifth cycle: sel=1, rd_en=1, wr_en=1;

[0090] Sixth cycle: sel=0, rd_en=0, wr_en=1;

[0091] …

[0092] End stage: last cycle: sel = 0, rd_en = 1, wr_en = 0;

[0093] The control signal in the cycle stage has a cycle from the fourth cycle to the sixth cycle.

[0094] Furthermore, when the data selection signal sel is 0, the control selector selects the convolution result to the intermediate FIFO buffer unit; when the data selection signal sel is 1, the control selector selects the sum of the convolution result and the frontmost data in the intermediate FIFO buffer unit to the intermediate FIFO buffer unit;

[0095] When the FIFO read enable signal rd_en is 1, the intermediate FIFO buffer unit performs a read operation, outputs the front data of the intermediate FIFO buffer unit to the first adder, and inputs the convolution result and the read data into the selector through the adder. When the FIFO read enable signal rd_en is 0, the intermediate FIFO buffer unit does not read;

[0096] When the FIFO write enable signal wr_en is 1, the intermediate FIFO cache unit performs a write operation to write the input data into the intermediate FIFO cache unit. When the FIFO write enable signal wr_en is 0, the intermediate FIFO cache unit is not written. At this time, the write enable is negated ~ wr_en = 1 is valid, indicating that the output result of the first adder to the output end of the cache scheduling circuit unit is valid, that is, the first adder outputs the result to the memory storage module.

[0097] Specifically, the selection signal and the enable signal output by the control logic unit control the selector and the intermediate FIFO buffer unit in the following manner:

[0098] Initial stage: First cycle: sel=0, rd_en=0, wr_en=1, wr_en=0; at this time, the arithmetic logic unit outputs the convolution result ①, which is selected by the selector and written into the intermediate FIFO buffer unit;

[0099] Second cycle: sel=1, rd_en=1, wr_en=1, ~wr_en=0; at this time, the arithmetic logic unit outputs the convolution result ② and inputs it into the first adder, the intermediate FIFO buffer unit reads the stored data ① and the convolution result ②, adds them together through the first adder, and outputs ①+② to the selector, the selector selects the summed result ①+②, and the intermediate FIFO buffer unit writes the summed result ①+②;

[0100] The third cycle: sel=0, rd_en=0, wr_en=1, wr_en=0; at this time, the convolution result ③ output by the arithmetic logic unit is selected by the selector and written into the intermediate FIFO buffer unit;

[0101] Loop stage: fourth cycle: sel=0, rd_en=1, wr_en=0, wr_en=1; at this time, the convolution result ④ output by the arithmetic logic unit is input to the first adder, the intermediate FIFO buffer unit reads the stored data ①+② and the convolution result ④, adds them together through the first adder, and outputs ①+②+④ to the output end of the cache scheduling circuit unit;

[0102] Cycle 5: sel = 1, rd_en = 1, wr_en = 1, and wr_en = 0. At this time, the convolution result ⑤ output by the arithmetic logic unit is input to the first adder. The intermediate FIFO buffer unit reads the stored data ③ and the convolution result ⑤, adds them together via the first adder, and outputs ③ + ⑤ to the selector. The selector selects the summed result ③ + ⑤, and the intermediate FIFO buffer unit writes the summed result ③ + ⑤.

[0103] Sixth cycle: sel=0, rd_en=0, wr_en=1, wr_en=0; at this time, the convolution result ⑥ output by the arithmetic logic unit is selected by the selector and written into the intermediate FIFO buffer unit;

[0104] …

[0105] Ending stage: Last cycle: sel = 0, rd_en = 1, wr_en = 0, ~wr_en = 1; at this time, the convolution result ○m output by the arithmetic logic unit is input to the first adder, the intermediate FIFO buffer unit reads the stored frontmost data and the convolution result ○m, adds them together through the first adder, and outputs the frontmost data + ○m to the output end of the cache scheduling circuit unit; thus completing the first row convolution;

[0106] The control signal in the cycle stage has a cycle from the fourth cycle to the sixth cycle.

[0107] Furthermore, the memory storage module includes a memory storage unit, a third adder and a temporary buffer area;

[0108] The memory storage unit includes n parts, each part storing a partial product which is a convolution result of a row of elements, and is used to store or read the partial product according to the external instruction signal and the storage address; wherein the convolution kernel size is n*n;

[0109] The third adder is used to add the partial products read from the memory storage unit to the corresponding convolution results to obtain a complete convolution result and store it in a temporary buffer area.

[0110] Furthermore, the number of the third adders and temporary buffer areas is consistent with the number of convolution kernels, both of which are Cout. Each third adder adds each part of the convolution result of the corresponding convolution kernel and stores it in the corresponding temporary buffer area.

[0111] Furthermore, the external command signal includes a memory read command signal memory-rd and a memory write command signal memory-wr;

[0112] When memory-rd=1 and memory-wr=0, the memory storage unit sequentially reads the n parts of data after the element is convolved with each convolution kernel and inputs them to the third adder; the third adder performs an addition operation and inputs them into the corresponding temporary buffer area;

[0113] When memory-rd=0 and memory-wr=1, the data in the temporary buffer is stored in memory;

[0114] When memory-rd=0 and memory-wr=0, the memory storage unit does not operate.

[0115] Specifically, since only one instruction can be executed, memory-rd=1 and memory-wr=1 will not appear at the same time.

[0116] Specifically, the memory input end also includes an address interface for receiving a storage address, where Addr is the partial product address line of the first convolution kernel, addr+ww is the partial product address line of the second convolution kernel, addr+2ww is the partial product address line of the third convolution kernel, ..., addr+(Cout-1)addr is the partial product address line of the Cout-th convolution kernel; addr0, add1, and addr2 are the partial product address lines of each convolution kernel respectively;

[0117] The convolution results output by the output end of the Cout convolution kernel after convolution in the intermediate logic operation unit are stored in the memory storage unit according to the storage address.

[0118] When memory-rd=1 and memory-wr=0, the memory storage unit reads the three parts of memory data of the first convolution kernel (1,1) with addresses addr0, addr1, and addr2 in sequence to the third adder 1, adds the data to obtain the convolution kernel sum1 and stores it in the temporary buffer reg1; reads the three parts of memory data of the second convolution kernel (1,1) with addresses addr0+ww, addr1+ww, and addr2+ww to the third adder 2, adds the data to the temporary buffer reg2 after addition by the third adder 2, ..., reads the three parts of memory data of the Coutth convolution kernel (1,1) with addresses addr0+(Cout-1)ww, addr1+(Cout-1)ww, and addr2+(Cout-1)ww to the third adder Cout, and stores the result in the temporary buffer reg_cout. In the next read instruction, in another cycle, the three parts of each element of the Cout convolution kernel are added simultaneously; the three parts of the memory data of the first convolution kernel (1,2) with addresses addr0, addr1, and addr2 are read to the third adder 1, and so on.

[0119] When memory-rd = 0 and memory-wr = 1, the data in temporary buffer reg1 is stored in memory memory. The sum of the data (1,1) by the third adder 1 is stored in temporary buffer reg1, with the input memory location address being addr0. The sum of the data by the third adder 2 is stored in temporary buffer reg2, with the input memory location address being addr0 + WW. ... The sum of the data by the third adder Cout is stored in reg_cout, with the input memory location address being addr0 + (Cout - 1) * WW. In the next storage command, that is, in a different cycle, the sum of the data (1,2) by the third adder 1 is stored in reg1, with the input memory location address being addr0, and so on.

[0120] When memory-rd=0 and memory-wr=0, the memory does not perform any operation.

[0121] WW is the word length, usually 4 bytes or 8 bytes.

[0122] In one possible implementation, the number of intermediate cache scheduling circuit units is 1, the number of convolution kernels is 1, the number of third adders and temporary cache areas is 1, the memory storage unit has 3 partial products, the intermediate cache scheduling circuit unit receives the elements and weights of each operation cycle in time-sharing, and outputs the calculation results datah of each operation cycle in sequence from the output end; the memory storage unit stores the calculation results datah of each operation cycle output by the output end of the intermediate cache scheduling circuit unit in sequence; according to the memory read instruction signal, the memory storage unit reads the partial products of the corresponding elements in each part into the temporary cache area, and according to the memory write instruction signal, the memory storage unit writes the data in the temporary cache area back to the memory storage unit; wherein h is the ordinal number of the calculation cycle; and each operation cycle includes the sum of the convolution results corresponding to three groups of adjacent elements.

[0123] Specifically, such as Figure 1 As shown, when the number of intermediate cache scheduling single-way units is 1, the intermediate cache scheduling circuit device supporting the in-memory CNN adopts a partial multiplexing strategy; the calculation result data of one operation cycle represents 1 / 3 of the complete convolution result of the input array and the convolution kernel at each step size, and the complete convolution result is input to the memory storage unit and stored in the memory storage unit;

[0124] When the elements of a row in the input array are retrieved in sequence, the elements of the second row of the input array and the weights of the corresponding row of the convolution kernel are input, and the results are input into the memory storage unit in sequence; when the convolution of the three rows of elements is completed, the data of the corresponding columns in the memory storage unit are summed to obtain the convolution result of the first row of the input array and the convolution kernel in the row direction;

[0125] The convolution kernel is shifted down one grid in the column direction with a step size of 1. The convolution result of the second row of the input array and the convolution kernel in the row direction is obtained according to the above method; and so on.

[0126] Specifically, when a partial reuse strategy is adopted, the data that can be reused includes:

[0127] input_map*cin*kernel_row+cin*kernel_size*cout*input_map;

[0128] When using a 3*3 convolution kernel, it can be simplified to:

[0129] input_map*cin*3+cin*9*cout*input_map;

[0130] Among them, input_map is the size of the input array, cin is the number of channels of input elements and weights, kernel_row is the convolution kernel row, kernel_size is the size of the convolution kernel; cout is the number of output channels of the input array and convolution kernel;

[0131] The data that can be reused during the storage phase includes:

[0132] 1*1*cout*output_map*kernel_row;

[0133] When using a 3*3 convolution kernel, it can be simplified to:

[0134] cout*output_map*3;

[0135] Among them, output_map is the size of the new array obtained after convolution of the input array and the convolution kernel.

[0136] In another possible implementation, the number of the intermediate cache scheduling circuit units is 3, and the number of convolution kernels is Cout; the number of the third adder and the temporary cache area is Cout; the input ends of the three intermediate cache scheduling circuit units respectively input the three rows of weights and the elements of the first row of the convolution kernel and the three intermediate cache scheduling circuit units are completed in parallel, and the corresponding output ends sequentially output the calculation results datah1, datah2, and datah3 of each corresponding operation cycle; the calculation results datah1, datah2, and datah3 are respectively stored in the three partial products of the memory storage unit, and are arranged in order according to the ordinal number of the convolution kernel. According to the memory read instruction signal, the memory storage unit reads the partial products of the corresponding elements in each part of each convolution kernel to the corresponding temporary cache area; according to the memory write instruction signal and the storage address, the memory storage unit writes the data in the temporary storage back to the memory storage unit; wherein, 1-3 is the ordinal number of the intermediate cache scheduling circuit unit.

[0137] Specifically, when the number of intermediate cache scheduling single-way units is 3, the intermediate cache scheduling circuit device supporting the in-memory CNN adopts a full multiplexing strategy; when performing convolution on the first row of elements, only one intermediate cache scheduling circuit unit is used, and its input end inputs the first row of elements and the first row of weights; when performing convolution on the second row of elements, two intermediate cache scheduling circuit units are used, the input end of the first intermediate cache scheduling circuit unit inputs the second row of elements and the first row of weights, and the input end of the second intermediate cache scheduling circuit unit inputs the second row of elements and the second row of weights; when performing convolution on the third row of elements, three intermediate cache scheduling circuit units are used, the input end of the first intermediate cache scheduling circuit unit inputs the third row of elements and the first row of weights, and the input end of the second intermediate cache scheduling circuit unit inputs the third row of elements and the first row of weights. The third row of elements and the second row of weights are input to the input end of the intermediate cache scheduling circuit unit, and the third row of elements and the third row of weights are input to the input end of the third intermediate cache scheduling circuit unit; starting from the third row of elements, each row of elements is simultaneously convolved with the three rows of weights by the three intermediate cache scheduling circuit units; when convolving the second-to-last row of elements, two intermediate cache scheduling circuit units are used, the second-to-last row of elements and the third row of weights are input to the input end of the first intermediate cache scheduling circuit unit, and the second row of elements and the second row of weights are input to the input end of the second intermediate cache scheduling circuit unit; when convolving the first-to-last row of elements, only one intermediate cache scheduling circuit unit is used, and its input end inputs the first-to-last row of elements and the third row of weights;

[0138] When performing convolution on each row of elements, the intermediate cache circuit unit used simultaneously outputs the calculation results of each corresponding operation cycle to the memory storage unit, that is, the convolution of a row of elements of the input array and the weights of each row of the convolution kernel is calculated at the same time, and there is no need to repeatedly read the elements of this row of the input array; the data of the corresponding column is added in the memory storage unit to obtain the convolution result of the input array and the convolution kernel.

[0139] Specifically, when a full reuse strategy is adopted, the data that can be reused includes:

[0140] input_map*cin+cin*kernel_size*cout*input_map*kernel_row;

[0141] When using a 3*3 convolution kernel, it can be simplified to:

[0142] input_map*cin+cin*9*cout*input_map*3;

[0143] The data that can be reused during the storage phase includes:

[0144] 1*1*cout*output_map;

[0145] When using a 3*3 convolution kernel, it can be simplified to:

[0146] cout*output_map*3.

[0147] Specifically, the full reuse strategy reuses the input array more completely than the partial reuse strategy. The full reuse strategy only needs to read the row elements of the input array once, reducing the power consumption of repeated reads after the calculation is complete. Furthermore, parallel computing can reduce the computation time in the arithmetic logic unit. The partial reuse strategy requires multiple reads of the row elements of the input array, repeatedly reading the row elements after calculating a row, which increases the power consumption of the read distance compared to the full reuse strategy. However, full reuse also has disadvantages compared to the partial reuse strategy: the intermediate FIFO buffer unit size of the full reuse strategy requires kernel_size*1*1*cout, which is approximately equal to 9*cout for a 3*3 convolution kernel. In contrast, the intermediate FIFO buffer unit size of the partial reuse strategy only requires kernel_row*1*1*cout, which is approximately equal to 3*cout for a 3*3 convolution kernel. Therefore, the capacity area of ​​the intermediate FIFO buffer unit in the full reuse strategy is three times that of the partial reuse strategy. In specific implementation, a trade-off between area and power consumption can be made based on the needs of the application circuit.

[0148] Specifically, the above intermediate cache scheduling circuit device supporting in-memory CNN only describes a convolution cache device for a convolution kernel and an input array, and the actual application is not limited to one convolution kernel, such as Figure 1 and Figure 2 As shown, multiple convolution kernels can share an input array. When there are cout convolution kernels, the input array and the output channel of cout convolution kernels are cout, that is, there are cout intermediate cache scheduling circuit devices.

[0149] Specifically, the size of the input array is input_col*incup_row*cin, the convolution kernel size is kernel_row*kernel_rcol*cin, cout convolution kernels share 1 input array, and the new array is output_col*outcup_row*cout.

[0150] Compared to existing technologies, the scheduling scheme provided in this embodiment reduces the power consumption and latency associated with multiple data accesses by adding an intermediate cache. It leverages in-memory technology to significantly increase the parallelism of memory accesses, thereby increasing data computation efficiency through parallelization. Furthermore, this scheme simplifies the scheduling logic for the intermediate cache. By implementing a first-in, first-out (FIFO) strategy and simple loop logic control, it reduces the control complexity associated with using SRAM or a register file as the intermediate cache.

[0151] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0152] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A memory based on in-memory CNN intermediate cache scheduling, characterized in that: It includes a control logic unit, a cache scheduling circuit unit and a memory storage module; the cache scheduling circuit unit includes an arithmetic logic unit, a first adder, a selector and an intermediate FIFO cache unit; The control logic unit is used to output a selection signal and an enable signal according to a clock signal and an externally input valid flag signal; The arithmetic logic unit is used to perform a convolution operation on the elements of the input array and the weights of the convolution kernel, and output the corresponding convolution result to the selector; The first adder is configured to add the convolution result and the data output by the intermediate FIFO buffer unit based on an enable signal, and output the summed result to the selector or output the summed result to the memory storage module based on an enable negation signal; The selector is configured to select and output the convolution result or the summation result based on the selection signal; The intermediate FIFO buffer unit is configured to select whether to write the data output by the selector into the intermediate FIFO buffer unit and to select whether to output the stored data according to the enable signal; The memory storage module is used to store and calculate the addition result output by the first adder according to the external instruction signal and the storage address; The enable signal includes a FIFO read enable signal and a FIFO write enable signal; The FIFO read enable signal is used to control the intermediate FIFO buffer unit to read data; The FIFO write enable signal is used to control the intermediate FIFO buffer unit to write data or output data to the output end of the intermediate buffer scheduling circuit unit; Set the selection signal to sel, the FIFO read enable signal to rd_en, and the FIFO write enable signal to wr_en. When used for 3*3 convolution kernel operations and the valid flag signal is 1, the selection signal and enable signal output by the control logic unit include three stages: Initial stage: first cycle: sel = 0, rd_en = 0, wr_en = 1; Second cycle: sel=1, rd_en=1, wr_en=1; The third cycle: sel = 0, rd_en = 0, wr_en = 1; Cycle phase: fourth cycle: sel = 0, rd_en = 1, wr_en = 0; Fifth cycle: sel=1, rd_en=1, wr_en=1; Sixth cycle: sel=0, rd_en=0, wr_en=1; … Ending stage: last cycle: sel=0, rd_en=1, wr_en=0.

2. The memory based on in-memory CNN intermediate cache scheduling according to claim 1, characterized in that The memory storage module includes a memory storage unit, a third adder and a temporary buffer area; The memory storage unit includes n parts, each part storing a partial product which is a convolution result of a row of elements, and is used to store or read the partial product according to the external instruction signal and the storage address; The convolution kernel size is n*n; The third adder is used to add the partial products read from the memory storage unit to the corresponding convolution results to obtain a complete convolution result and store it in a temporary buffer area.

3. The memory based on in-memory CNN intermediate cache scheduling according to claim 2, characterized in that The number of the third adders and temporary buffer areas is consistent with the number of convolution kernels, both of which are Cout. Each third adder adds each part of the convolution result of the corresponding convolution kernel and stores it in the corresponding temporary buffer area.

4. The memory based on in-memory CNN intermediate cache scheduling according to claim 3, characterized in that The external command signal includes a memory read command signal memory-rd and a memory write command signal memory-wr; When memory-rd=1 and memory-wr=0, the memory storage unit sequentially reads the n parts of data after the element is convolved with each convolution kernel to the third adder; The third adder performs addition operation and inputs the result into the corresponding temporary buffer area; When memory-rd=0 and memory-wr=1, the data in the temporary buffer is stored in the memory storage unit; When memory-rd=0 and memory-wr=0, the memory storage unit does not operate.

5. The memory based on in-memory CNN intermediate cache scheduling according to claim 4, characterized in that When the cache scheduling circuit performs an operation with an n*n convolution kernel size, the storage size of the intermediate FIFO cache unit is n.

6. The memory based on in-memory CNN intermediate cache scheduling according to claim 5, characterized in that When the selection signal sel is 0, the control selector selects the convolution result to the intermediate FIFO buffer unit; when the selection signal sel is 1, the control selector selects the sum of the convolution result and the frontmost data in the intermediate FIFO buffer unit to the intermediate FIFO buffer unit; When the FIFO read enable signal rd_en is 1, the intermediate FIFO buffer unit performs a read operation, outputs the front data of the intermediate FIFO buffer unit to the first adder, and inputs the convolution result and the data read by the read operation into the selector through the adder. When the FIFO read enable signal rd_en is 0, the intermediate FIFO buffer unit does not read; When the FIFO write enable signal wr_en is 1, the intermediate FIFO cache unit performs a write operation to write the input data into the intermediate FIFO cache unit. When the FIFO write enable signal wr_en is 0, the intermediate FIFO cache unit is not written. At this time, the write enable is negated ~ wr_en = 1 is valid, indicating that the output result of the first adder to the output end of the cache scheduling circuit unit is valid, that is, the first adder outputs the result to the memory storage module.

7. The memory based on in-memory CNN intermediate cache scheduling according to claim 1 or 6, characterized in that: The number of convolution kernels is 1, the number of the third adder and the temporary cache area are both 1, the memory storage unit has 3 partial products, and the memory storage unit sequentially stores the calculation results datah of each operation cycle output by the output end of the intermediate cache scheduling circuit unit; according to the memory read instruction signal, the memory storage unit reads the partial products of the corresponding elements in each part to the temporary cache area, and according to the memory write instruction signal, the memory storage unit writes the data in the temporary cache area back to the memory storage unit; wherein h is the ordinal number of the operation cycle; and each operation cycle includes the sum of the convolution results corresponding to the convolution of three groups of adjacent elements.

8. The memory based on in-memory CNN intermediate cache scheduling according to claim 1 or 6, characterized in that: The number of the intermediate cache scheduling circuit units is 3, and the number of convolution kernels is Cout; the number of the third adder and the temporary cache area is Cout, and the input ends of the three intermediate cache scheduling circuit units respectively input the three rows of weights of the convolution kernel and the elements of the first row. The three intermediate cache scheduling circuit units are completed in parallel, and the corresponding output ends sequentially output the calculation results datah1, datah2, and datah3 of the corresponding operation cycles; the calculation results datah1, datah2, and datah3 are respectively stored in the three partial products of the memory storage unit, and are arranged in order according to the ordinal number of the convolution kernel. According to the memory read instruction signal, the memory storage unit reads the partial products of the corresponding elements in each part of each convolution kernel to the corresponding temporary cache area; according to the memory write instruction signal and the storage address, the memory storage unit writes the data in the temporary storage back to the memory storage unit; wherein, 1-3 is the ordinal number of the intermediate cache scheduling circuit unit.

Images