An intermediate cache scheduling method supporting in-memory CNN

Abstract

The present invention relates to an intermediate cache scheduling method supporting in-memory CNNs, belonging to the field of multiplexed CNN scheduling operation technology, and solves the problems of high energy consumption and high latency caused by memory access and data transfer in the prior art. A method for scheduling an intermediate cache supporting in-memory CNNs includes the following steps: performing a convolution operation on the elements of an input array and the weights of a convolution kernel to obtain a convolution result; outputting the selection result according to a selection signal; the selection result includes a convolution result or a summation result; wherein the summation result is the sum of the convolution result and the data output by the intermediate FIFO cache unit; selecting whether to write the selection result to the intermediate FIFO cache unit and whether to output the frontmost data stored in the intermediate FIFO cache unit according to an enable signal; wherein the selection signal and the enable signal are obtained based on a clock signal and an external input valid flag signal. This method realizes the simplification of CNN scheduling through FIFO cache.

Description

Technical Field

[0001] The present invention relates to the field of multiplexed CNN scheduling operation technology, and in particular to an intermediate cache scheduling method supporting in-memory CNN. Background Art

[0002] In the era of big data, existing computing architectures face increased energy consumption and latency when faced with data-intensive applications due to large amounts of data transfer and bandwidth limitations. This results in vector parallelism on data being constrained by memory read and write bandwidth. To address the energy consumption and latency caused by memory access and data transfer, in-memory computing breaks through the traditional von Neumann bottleneck, achieving the integration of memory and logic units and becoming one of the primary technical approaches for intelligent computing. Academia has proposed SRAM-based arithmetic and logic circuits to implement arithmetic and logic operations in memory. However, existing SRAM-based arithmetic and logic circuits have a relatively simple computational model, limiting their performance when operating between external inputs and memory units. Current in-memory computing technologies face challenges such as hardware resource reuse.

[0003] A convolutional neural network is a deep neural network with a convolutional architecture. This structure can reduce the amount of memory used by deep networks and the number of parameters, alleviating model overfitting. The convolution kernel is a crucial component of the convolutional layer. The kernel's essence is the feature extractor, whose primary function is to automatically extract deep information from the input signal. A convolutional layer consists of multiple feature planes, each of which is composed of multiple neurons. Each neuron is connected to a local region of the feature plane in the previous layer via a convolution kernel. The feature map of the previous layer is convolved with a learnable convolution kernel and then passed through an activation function to produce the output feature map. Each neuron is locally connected to neurons in the previous layer and shares weights. The output data is then deconvolved with the image using the shared weights and then a shared bias is added to each neuron. Without shared weights, each neuron would require a single convolution kernel and a single bias. Instead, each neuron is assigned the same convolution kernel and bias, significantly reducing the number of parameters and making it suitable for in-memory data reuse. Existing techniques for multiple data accesses result in significant memory loss and high latency, resulting in low data processing efficiency. Summary of the Invention

[0004] In view of the above analysis, an embodiment of the present invention aims to provide a partial reuse intermediate cache scheduling method supporting in-memory CNN, so as to solve the problems of energy consumption and delay caused by existing memory access and data transfer.

[0005] In one aspect, an embodiment of the present invention provides an intermediate cache scheduling method supporting an in-memory CNN, comprising the following steps:

[0006] Perform convolution operation on the elements of the input array and the convolution kernel weights to obtain the convolution result;

[0007] Outputting a selection result according to a selection signal; the selection result includes a convolution result or a summation result; wherein the summation result is the sum of the convolution result and the data output by the intermediate FIFO buffer unit;

[0008] Select whether to write the selection result into the intermediate FIFO buffer unit and select whether to output the front data stored in the intermediate FIFO buffer unit according to the enable signal;

[0009] Output the sum result based on the enable negation signal as the output result of a cache scheduling;

[0010] The selection signal and the enable signal are obtained based on a clock signal and an externally input valid flag signal.

[0011] Furthermore, the elements of the input array and the weights of the convolution kernel are determined according to the cache operation logic; the valid flag signal is used to control whether the elements and the weights of the convolution kernel are written; when the valid flag signal is 0, it is not written, and when it is 1, it is written.

[0012] Furthermore, when the intermediate cache scheduling method performs a 3*3 convolution kernel operation, the selection signal and the enable signal are obtained based on the clock signal and the valid flag signal of the external input, including:

[0013] When the valid flag signal is 1, it indicates that the weight of the element and convolution kernel is written; the selection signal and enable signal output by the control logic unit include three stages:

[0014] Initial stage: first cycle: sel = 0, rd_en = 0, wr_en = 1;

[0015] Second cycle: sel=1, rd_en=1, wr_en=1;

[0016] The third cycle: sel = 0, rd_en = 0, wr_en = 1;

[0017] Cycle phase: fourth cycle: sel = 0, rd_en = 1, wr_en = 0;

[0018] Fifth cycle: sel=1, rd_en=1, wr_en=1;

[0019] Sixth cycle: sel=0, rd_en=0, wr_en=1;

[0020] …

[0021] End stage: last cycle: sel = 0, rd_en = 1, wr_en = 0;

[0022] The control signal in the cycle stage has a cycle from the fourth cycle to the sixth cycle; among them, sel represents the selection signal, rd_en represents the FIFO read enable signal, and wr_en represents the FIFO write enable signal.

[0023] Furthermore, the outputting of a selection result according to the selection signal; the selection result including a convolution result or a summation result includes:

[0024] When the selection signal is 1, the convolution result is selected for output;

[0025] When the selection signal is 0, the selection outputs the sum result.

[0026] Furthermore, the enable signal includes a FIFO read enable signal and a FIFO write enable signal;

[0027] The FIFO read enable signal is used to control whether the intermediate FIFO buffer unit reads out data;

[0028] The FIFO write enable signal is used to control the intermediate FIFO buffer unit to write data or output data to the output end of the intermediate buffer scheduling circuit unit.

[0029] Furthermore, the FIFO read enable signal is used to control whether the intermediate FIFO buffer unit reads out data, including:

[0030] When the FIFO read enable signal is 1, the intermediate FIFO buffer unit reads out data;

[0031] When the FIFO read enable signal is 0, the intermediate FIFO buffer unit does not read data;

[0032] Furthermore, the FIFO write enable signal is used to control the intermediate FIFO buffer unit to write data or output data to the output end of the intermediate buffer scheduling circuit unit, including:

[0033] When the FIFO write enable signal is 1, the intermediate FIFO buffer unit writes the selection result;

[0034] When the FIFO write enable signal is 0, the intermediate FIFO buffer unit does not write the selection result, and the write enable signal is inverted to 1 to control data output to the output end of the intermediate buffer scheduling circuit unit.

[0035] Furthermore, the cache operation logic includes calling each column element in the input array in sequence by row, and obtaining several convolution results corresponding to the element based on the movement relationship between the element and the convolution kernel; wherein each element of the convolution kernel is a weight.

[0036] Furthermore, the convolution operation performed on the elements of the input array and the convolution kernel weights to obtain the convolution result includes calculating the convolution of one element and the weight each time to obtain a convolution result; and storing the result in an intermediate FIFO cache unit through a selection signal and an enable signal.

[0037] Furthermore, when the intermediate cache scheduling method performs a 3*3 convolution kernel operation, the convolution operation on the elements of the input array and the convolution kernel weights to obtain the convolution result further includes:

[0038] Calculate the convolution of an element in the first row of the input array and the corresponding weight to obtain a convolution result; store the result in the first intermediate FIFO buffer unit through the selection signal and the enable signal;

[0039] Calculate the convolution of an element in the second row of the input array and the corresponding weight to obtain a convolution result; store the result in the second intermediate FIFO buffer unit through the selection signal and the enable signal;

[0040] Calculate the convolution of an element in the third row of the input array and the corresponding weight to obtain a convolution result; store the result in the third intermediate FIFO buffer unit through the selection signal and the enable signal;

[0041] Among them, the convolution of the above three rows of elements and weights is performed simultaneously.

[0042] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0043] 1. This application is based on the form of an in-memory vector processing unit, and the convolution operation process adopts a row multiplexing method, which reduces the energy consumption and delay caused by memory access and data transfer. The intermediate cache scheduling method based on the intermediate cache scheduling circuit device can make full use of the configurability of the convolution vector and realize convolution support for parameters of arbitrary length in multiple cycles.

[0044] 2. The scheduling logic of the intermediate cache is simplified. Through the first-in-first-out FIFO strategy and simple loop logic control, the control complexity of using SRAM or register file as the intermediate cache is reduced.

[0045] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0047] Figure 1 This is a flow chart of the intermediate cache scheduling method of this application;

[0048] Figure 2 This is the intermediate cache circuit architecture diagram of this application;

[0049] Figure 3 Implement the model graph for the convolution acceleration module;

[0050] Figure 4 This is the overall architecture diagram of the intermediate cache scheduling of the CNN in this application;

[0051] Figure 5 To cache the logic calculation graph;

[0052] Figure 6 It is an existing SIMD calculation graph. DETAILED DESCRIPTION

[0053] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0054] A specific embodiment of the present invention discloses an intermediate cache scheduling method supporting in-memory CNN, comprising the following steps:

[0055] S110. Perform a convolution operation on the elements of the input array and the convolution kernel weights to obtain a convolution result; specifically, this can be achieved through an arithmetic logic unit.

[0056] S120. Output a selection result according to the selection signal; the selection result includes a convolution result or a summation result; wherein the summation result is the sum of the convolution result and the data output by the intermediate FIFO buffer unit.

[0057] Specifically, the outputting of the selection result according to the selection signal can be implemented by a selector; and the adding can be implemented by a first adder.

[0058] S130, selecting whether to write the selection result into the intermediate FIFO buffer unit and selecting whether to output the frontmost data stored in the intermediate FIFO buffer unit according to the enable signal; wherein the selection signal and the enable signal are obtained based on the clock signal and the valid flag signal of the external input.

[0059] Specifically, the control logic unit may obtain the selection signal and the enable signal based on the clock signal and the valid flag signal input externally.

[0060] S140 : Outputting a summation result based on the enable negation signal as an output result of a cache scheduling.

[0061] Specifically, the intermediate cache scheduling method of the present application is implemented based on an intermediate cache scheduling circuit device supporting in-memory CNN, such as Figure 1 As shown, it includes a control logic unit and a cache scheduling circuit unit; the cache scheduling unit includes an arithmetic logic unit, a first adder, a selector and an intermediate FIFO cache unit;

[0062] The control logic unit is used to output a selection signal and an enable signal according to a clock signal and an externally input valid flag signal;

[0063] The arithmetic logic unit is used to perform a convolution operation on the elements of the input array and the weights of the convolution kernel, and output the corresponding convolution result to the selector;

[0064] The first adder is configured to add the convolution result and the data output by the intermediate FIFO buffer unit based on an enable signal, and output the summed result to the selector or output the summed result to the output end of the cache scheduling circuit unit based on an enable negation signal;

[0065] The selector is configured to select and output the convolution result or the summation result based on the selection signal;

[0066] The intermediate FIFO buffer unit is used to select whether to write the data output by the selector into the intermediate FIFO buffer unit and to select whether to output the stored data according to the enable signal.

[0067] Specifically, the intermediate FIFO buffer unit is a first-in-first-out queue, which simplifies the scheduling strategy.

[0068] Specifically, such as Figure 3 As shown, the memory unit including the control logic unit and the intermediate cache scheduling circuit unit of the present application also includes a memory storage unit; the input array and the convolution kernel are first calculated in the intermediate cache scheduling circuit unit after entering the memory, and the operation logic is divided into a start stage, a loop stage and an end stage; the data value output by the intermediate cache scheduling circuit unit is input into the memory storage unit, which is stored by the memory storage unit and the corresponding column of the convolution kernel row size is added and then stored; that is, from the operation logic unit to the intermediate FIFO cache unit, and then entering the memory storage unit from the intermediate FIFO cache unit, there is no need to exit the entire memory unit, thereby reducing the bus bandwidth between the memory unit and the processor.

[0069] Specifically, the arithmetic logic unit includes cin multipliers and a second adder;

[0070] The i-th multiplier is used to perform a multiplication operation of Ai and Wi; wherein Ai is the i-th value in the element, and Wi is the i-th value in the weight; i∈[1,cin];

[0071] The second adder is used to obtain the sum of the output results of cin multipliers and output the sum result as the corresponding convolution result.

[0072] Further, such as Figure 4 As shown, the elements of the input arithmetic logic unit and the weights of the convolution kernel are determined according to the cache arithmetic logic; the valid flag signal is used to control whether the arithmetic logic unit writes the elements and the weights of the convolution kernel; when the valid flag signal is 0, it is not written, and when it is 1, it is written.

[0073] Specifically, the cache operation logic is to reduce the movement of elements A in the input array. An intermediate FIFO cache unit is selected to unilaterally change the convolution kernel weight W, thereby obtaining the convolution result of a row of elements in the input array and a row of weights of the convolution kernel.

[0074] Specifically, the rules for input elements and weights of cache operation logic are as follows:

[0075] Input values ​​for the first cycle: elements in the first row and first column and weights in the first row and first column;

[0076] Input values ​​for the second cycle: elements in the first row and second column and weights in the first row and second column;

[0077] Input values ​​for the third cycle: elements in the first row and second column and weights in the first row and first column;

[0078] Input values ​​for the fourth cycle: elements in the first row and third column and weights in the first row and third column;

[0079] Input values ​​for the fifth cycle: elements in the first row and third column and weights in the first row and second column;

[0080] Input values ​​for the sixth cycle: elements in the first row and third column and weights in the first row and first column;

[0081] …

[0082] Input value of the third-to-last period: elements in the first row and second-to-last column and weights in the first row and third column;

[0083] Input values ​​of the second-to-last period: elements in the first row and the second-to-last column and weights in the first row and the second column;

[0084] Input value of the last period: the element of the first row and the last column and the weight of the first row and the third column;

[0085] Starting from the seventh cycle, the element input value is shifted to the right by one every three cycles, and the weight is cycled in the same way as the input from the fourth cycle to the sixth cycle.

[0086] Furthermore, the cache operation logic includes calling each column element in the input array in sequence by row, and obtaining several convolution results corresponding to the element based on the movement relationship between the element and the convolution kernel; wherein each element of the convolution kernel is a weight.

[0087] Specifically, the input and convolution result output of the one logic operation unit are expressed as:

[0088]

[0089] Among them, A is an element and W is a weight.

[0090] Specifically, when the cache scheduling circuit performs an operation with a convolution kernel size of n*n, the storage size of the intermediate FIFO cache unit is n.

[0091] Furthermore, the enable signal includes a FIFO read enable signal and a FIFO write enable signal;

[0092] The FIFO read enable signal is used to control the intermediate FIFO buffer unit to read data;

[0093] The FIFO write enable signal is used to control the intermediate FIFO buffer unit to write data or output data to the output end of the intermediate buffer scheduling circuit unit.

[0094] Furthermore, the selection signal is set to sel, the FIFO read enable signal is set to rd_en, and the FIFO write enable signal is set to wr_en. When used for the operation of the 3*3 convolution kernel and the valid flag signal is 1, the selection signal and enable signal output by the control logic unit include three stages:

[0095] Initial stage: first cycle: sel = 0, rd_en = 0, wr_en = 1;

[0096] Second cycle: sel=1, rd_en=1, wr_en=1;

[0097] The third cycle: sel = 0, rd_en = 0, wr_en = 1;

[0098] Cycle phase: fourth cycle: sel = 0, rd_en = 1, wr_en = 0;

[0099] Fifth cycle: sel=1, rd_en=1, wr_en=1;

[0100] Sixth cycle: sel=0, rd_en=0, wr_en=1;

[0101] …

[0102] End stage: last cycle: sel = 0, rd_en = 1, wr_en = 0;

[0103] The control signal in the cycle stage has a cycle from the fourth cycle to the sixth cycle.

[0104] Furthermore, the outputting of a selection result according to the selection signal; the selection result including a convolution result or a summation result includes:

[0105] When the selection signal is 1, the convolution result is selected for output;

[0106] When the selection signal is 0, the selection outputs the sum result.

[0107] Specifically, when the data selection signal sel is 0, the control selector selects the convolution result to the intermediate FIFO buffer unit; when the data selection signal sel is 1, the control selector selects the sum of the convolution result and the frontmost data in the intermediate FIFO buffer unit to the intermediate FIFO buffer unit;

[0108] Furthermore, the FIFO read enable signal is used to control whether the intermediate FIFO buffer unit reads out data, including:

[0109] When the FIFO read enable signal is 1, the intermediate FIFO buffer unit reads out data;

[0110] When the FIFO read enable signal is 0, the intermediate FIFO buffer unit does not read data;

[0111] Specifically, when the FIFO read enable signal rd_en is 1, the intermediate FIFO buffer unit performs a read operation, outputs the front data of the intermediate FIFO buffer unit to the first adder, and inputs the convolution result and the read data into the selector through the adder. When the FIFO read enable signal rd_en is 0, the FIFO does not read;

[0112] Furthermore, the FIFO write enable signal is used to control the intermediate FIFO buffer unit to write data or output data to the output end of the intermediate buffer scheduling circuit unit, including:

[0113] When the FIFO write enable signal is 1, the intermediate FIFO buffer unit writes the selection result;

[0114] When the FIFO write enable signal is 0, the intermediate FIFO buffer unit does not write the selection result, and the write enable signal is inverted to 1 to control data output to the output end of the intermediate buffer scheduling circuit unit.

[0115] Specifically, when the FIFO write enable signal wr_en is 1, the intermediate FIFO cache unit performs a write operation to write the input data, that is, the convolution result or the sum of the convolution result and the data read out by the intermediate FIFO cache unit, into the intermediate FIFO cache unit. Finally, when the FIFO write enable signal wr_en is 0, the intermediate FIFO cache unit is not written. At this time, the write enable is inverted ~ wr_en = 1 is valid, indicating that the output result of the first adder to the output end of the cache scheduling circuit unit is valid.

[0116] Specifically, the selection signal and the enable signal output by the control logic unit control the selector and the intermediate FIFO buffer unit in the following manner:

[0117] Initial stage: First cycle: sel=0, rd_en=0, wr_en=1, wr_en=0; at this time, the arithmetic logic unit outputs the convolution result ①, which is selected by the selector and written into the intermediate FIFO buffer unit;

[0118] Second cycle: sel=1, rd_en=1, wr_en=1, ~wr_en=0; at this time, the arithmetic logic unit outputs the convolution result ② and inputs it into the first adder, the intermediate FIFO buffer unit reads the stored data ① and the convolution result ②, adds them together through the first adder, and outputs ①+② to the selector, the selector selects the summed result ①+②, and the intermediate FIFO buffer unit writes the summed result ①+②;

[0119] The third cycle: sel=0, rd_en=0, wr_en=1, wr_en=0; at this time, the convolution result ③ output by the arithmetic logic unit is selected by the selector and written into the intermediate FIFO buffer unit;

[0120] Loop stage: fourth cycle: sel=0, rd_en=1, wr_en=0, wr_en=1; at this time, the convolution result ④ output by the arithmetic logic unit is input to the first adder, the intermediate FIFO buffer unit reads the stored data ①+② and the convolution result ④, adds them together through the first adder, and outputs ①+②+④ to the output end of the cache scheduling circuit unit;

[0121] Cycle 5: sel = 1, rd_en = 1, wr_en = 1, and wr_en = 0. At this time, the convolution result ⑤ output by the arithmetic logic unit is input to the first adder. The intermediate FIFO buffer unit reads the stored data ③ and the convolution result ⑤, adds them together via the first adder, and outputs ③ + ⑤ to the selector. The selector selects the summed result ③ + ⑤, and the intermediate FIFO buffer unit writes the summed result ③ + ⑤.

[0122] Sixth cycle: sel=0, rd_en=0, wr_en=1, wr_en=0; at this time, the convolution result ⑥ output by the arithmetic logic unit is selected by the selector and written into the intermediate FIFO buffer unit;

[0123] …

[0124] Ending stage: Last cycle: sel = 0, rd_en = 1, wr_en = 0, ~wr_en = 1; at this time, the convolution result ○m output by the arithmetic logic unit is input to the first adder, the intermediate FIFO buffer unit reads the stored frontmost data and the convolution result ○m, adds them together through the first adder, and outputs the frontmost data + ○m to the output end of the cache scheduling circuit unit; thus completing the first row convolution;

[0125] The control signal in the cycle stage has a cycle from the fourth cycle to the sixth cycle.

[0126] In one possible implementation, performing a convolution operation on the elements of the input array and the convolution kernel weights to obtain a convolution result includes calculating the convolution of one element and the weight each time to obtain a convolution result; and storing the result in an intermediate FIFO cache unit through a selection signal and an enable signal.

[0127] The number of intermediate cache scheduling circuit units is 1, and the intermediate cache scheduling circuit unit receives the elements and weights of each operation cycle in time-sharing, and outputs the calculation results datah of each operation cycle in sequence from the output end; wherein h is the ordinal number of the calculation cycle; and in each operation cycle, the convolution results corresponding to three groups of adjacent elements are included.

[0128] Specifically, such as Figure 1 As shown, when the number of intermediate cache scheduling single-way units is 1, the intermediate cache scheduling circuit device supporting the in-memory CNN adopts a partial multiplexing strategy; the calculation result data of one operation cycle represents 1 / 3 of the complete convolution result of the input array and the convolution kernel at each step size, and the complete convolution result is input to the memory storage unit and stored in the memory storage unit;

[0129] When the elements of a row in the input array are retrieved in sequence, the elements of the second row of the input array and the weights of the corresponding row of the convolution kernel are input, and the results are input into the memory storage unit in sequence; when the convolution of the three rows of elements is completed, the data of the corresponding columns in the memory storage unit are summed to obtain the convolution result of the first row of the input array and the convolution kernel in the row direction;

[0130] The convolution kernel is shifted down one grid in the column direction with a step size of 1. The convolution result of the second row of the input array and the convolution kernel in the row direction is obtained according to the above method; and so on.

[0131] Specifically, when a partial reuse strategy is adopted, the data that can be reused includes:

[0132] input_map*cin*kernel_row+cin*kernel_size*cout*input_map;

[0133] When using a 3*3 convolution kernel, it can be simplified to:

[0134] input_map*cin*3+cin*9*cout*input_map;

[0135] Among them, input_map is the size of the input array, cin is the number of channels of input elements and weights, kernel_row is the convolution kernel row, kernel_size is the size of the convolution kernel; cout is the number of output channels of the input array and convolution kernel;

[0136] The data that can be reused during the storage phase includes:

[0137] 1*1*cout*output_map*kernel_row;

[0138] When using a 3*3 convolution kernel, it can be simplified to:

[0139] cout*output_map*3;

[0140] Among them, output_map is the size of the new array obtained after convolution of the input array and the convolution kernel.

[0141] In another possible implementation, when the intermediate cache scheduling method performs a 3*3 convolution kernel operation, performing a convolution operation on the elements of the input array and the convolution kernel weights to obtain a convolution result further includes:

[0142] Calculate the convolution of an element in the first row of the input array and the corresponding weight to obtain a convolution result; store the result in the first intermediate FIFO buffer unit through the selection signal and the enable signal;

[0143] Calculate the convolution of an element in the second row of the input array and the corresponding weight to obtain a convolution result; store the result in the second intermediate FIFO buffer unit through the selection signal and the enable signal;

[0144] Calculate the convolution of an element in the third row of the input array and the corresponding weight to obtain a convolution result; store the result in the third intermediate FIFO buffer unit through the selection signal and the enable signal;

[0145] Among them, the convolution of the above three rows of elements and weights is performed simultaneously.

[0146] The number of the intermediate cache scheduling circuit units is 3; the input ends of the three intermediate cache scheduling circuit units respectively input the sum of the three rows of weights of the convolution kernel and the elements of the first row, and the three intermediate cache scheduling circuit units are completed in parallel, and the corresponding output ends sequentially output the calculation results datah1, datah2, and datah3 of the corresponding operation cycles; among them, 1-3 are the ordinal numbers of the intermediate cache scheduling circuit units.

[0147] Specifically, when the number of intermediate cache scheduling single-way units is 3, the intermediate cache scheduling circuit device supporting the in-memory CNN adopts a full multiplexing strategy; when performing convolution on the first row of elements, only one intermediate cache scheduling circuit unit is used, and its input end inputs the first row of elements and the first row of weights; when performing convolution on the second row of elements, two intermediate cache scheduling circuit units are used, the input end of the first intermediate cache scheduling circuit unit inputs the second row of elements and the first row of weights, and the input end of the second intermediate cache scheduling circuit unit inputs the second row of elements and the second row of weights; when performing convolution on the third row of elements, three intermediate cache scheduling circuit units are used, the input end of the first intermediate cache scheduling circuit unit inputs the third row of elements and the first row of weights, and the input end of the second intermediate cache scheduling circuit unit inputs the third row of elements and the first row of weights. The third row of elements and the second row of weights are input to the input end of the intermediate cache scheduling circuit unit, and the third row of elements and the third row of weights are input to the input end of the third intermediate cache scheduling circuit unit; starting from the third row of elements, each row of elements is simultaneously convolved with the three rows of weights by the three intermediate cache scheduling circuit units; when convolving the second-to-last row of elements, two intermediate cache scheduling circuit units are used, the second-to-last row of elements and the third row of weights are input to the input end of the first intermediate cache scheduling circuit unit, and the second row of elements and the second row of weights are input to the input end of the second intermediate cache scheduling circuit unit; when convolving the first-to-last row of elements, only one intermediate cache scheduling circuit unit is used, and its input end inputs the first-to-last row of elements and the third row of weights;

[0148] When performing convolution on each row of elements, the intermediate cache circuit unit used simultaneously outputs the calculation results of each corresponding operation cycle to the memory storage unit, that is, the convolution of a row of elements of the input array and the weights of each row of the convolution kernel is calculated at the same time, and there is no need to repeatedly read the elements of this row of the input array; the data of the corresponding column is added in the memory storage unit to obtain the convolution result of the input array and the convolution kernel.

[0149] Specifically, when a full reuse strategy is adopted, the data that can be reused includes:

[0150] input_map*cin+cin*kernel_size*cout*input_map*kernel_row;

[0151] When using a 3*3 convolution kernel, it can be simplified to:

[0152] input_map*cin+cin*9*cout*input_map*3;

[0153] The data that can be reused during the storage phase includes:

[0154] 1*1*cout*output_map;

[0155] When using a 3*3 convolution kernel, it can be simplified to:

[0156] cout*output_map*3.

[0157] Specifically, the full reuse strategy reuses the input array more completely than the partial reuse strategy. The full reuse strategy only needs to read the row elements of the input array once, reducing the power consumption of repeated reads after the calculation is complete. Furthermore, parallel computing can reduce the computation time in the arithmetic logic unit. The partial reuse strategy requires multiple reads of the row elements of the input array, repeatedly reading the row elements after calculating a row, which increases the power consumption of the read distance compared to the full reuse strategy. However, full reuse also has disadvantages compared to the partial reuse strategy: the intermediate FIFO buffer unit size of the full reuse strategy requires kernel_size*1*1*cout, which is approximately equal to 9*cout for a 3*3 convolution kernel. In contrast, the intermediate FIFO buffer unit size of the partial reuse strategy only requires kernel_row*1*1*cout, which is approximately equal to 3*cout for a 3*3 convolution kernel. Therefore, the capacity area of ​​the intermediate FIFO buffer unit in the full reuse strategy is three times that of the partial reuse strategy. In specific implementation, a trade-off between area and power consumption can be made based on the needs of the application circuit.

[0158] Specifically, the above intermediate cache scheduling circuit device supporting in-memory CNN only describes a convolution cache device for a convolution kernel and an input array, and the actual application is not limited to one convolution kernel, such as Figure 1 and Figure 2 As shown, multiple convolution kernels can share an input array. When there are cout convolution kernels, the input array and the output channel of cout convolution kernels are cout, that is, there are cout intermediate cache scheduling circuit devices.

[0159] Specifically, the size of the input array is input_col*incup_row*cin, the convolution kernel size is kernel_row*kernel_rcol*cin, cout convolution kernels share 1 input array, and the new array is output_col*outcup_row*cout.

[0160] Compared to existing technologies, the scheduling scheme provided in this embodiment reduces the power consumption and latency associated with multiple data accesses by adding an intermediate cache. It leverages in-memory technology to significantly increase the parallelism of memory accesses, thereby increasing data computation efficiency through parallelization. Furthermore, this scheme simplifies the scheduling logic for the intermediate cache. By implementing a first-in, first-out (FIFO) strategy and simple loop logic control, it reduces the control complexity associated with using SRAM or a register file as the intermediate cache.

[0161] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0162] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A method for scheduling intermediate caches supporting in-memory CNNs, characterized in that: The following steps are involved: Perform convolution operation on the elements of the input array and the convolution kernel weights to obtain the convolution result; Outputting a selection result according to a selection signal; the selection result includes a convolution result or a summation result; wherein the summation result is the sum of the convolution result and the data output by the intermediate FIFO buffer unit; Select whether to write the selection result into the intermediate FIFO buffer unit and select whether to output the front data stored in the intermediate FIFO buffer unit according to the enable signal; Output the sum result based on the enable negation signal as the output result of a cache scheduling; Wherein, the selection signal and the enable signal are obtained based on the clock signal and the valid flag signal input externally; When the intermediate cache scheduling method performs a 3*3 convolution kernel operation, the selection signal and the enable signal are obtained based on the clock signal and the valid flag signal of the external input, including: When the valid flag signal is 1, it indicates that the weight of the element and convolution kernel is written; the selection signal and enable signal output by the control logic unit include three stages: Initial stage: first cycle: sel = 0, rd_en = 0, wr_en = 1; Second cycle: sel=1, rd_en=1, wr_en=1; The third cycle: sel = 0, rd_en = 0, wr_en = 1; Cycle phase: fourth cycle: sel = 0, rd_en = 1, wr_en = 0; Fifth cycle: sel=1, rd_en=1, wr_en=1; Sixth cycle: sel=0, rd_en=0, wr_en=1; … End stage: last cycle: sel = 0, rd_en = 1, wr_en = 0; The control signal in the cycle stage has a cycle from the fourth cycle to the sixth cycle; among them, sel represents the selection signal, rd_en represents the FIFO read enable signal, and wr_en represents the FIFO write enable signal.

2. The intermediate cache scheduling method supporting in-memory CNN according to claim 1, characterized in that: The elements of the input array and the weights of the convolution kernel are determined according to the cache operation logic; the valid flag signal is used to control whether the elements and the weights of the convolution kernel are written; when the valid flag signal is 0, it is not written, and when it is 1, it is written.

3. The intermediate cache scheduling method supporting in-memory CNN according to claim 1, characterized in that: Outputting a selection result according to a selection signal; wherein the selection result includes a convolution result or a summation result, includes: When the selection signal is 1, the convolution result is selected for output; When the selection signal is 0, the selection outputs the sum result.

4. The intermediate cache scheduling circuit device supporting in-memory CNN according to claim 3, characterized in that: The enable signal includes a FIFO read enable signal and a FIFO write enable signal; The FIFO read enable signal is used to control whether the intermediate FIFO buffer unit reads out data; The FIFO write enable signal is used to control the intermediate FIFO buffer unit to write data or output data to the output end of the intermediate buffer scheduling circuit unit.

5. The intermediate cache scheduling method supporting in-memory CNN according to claim 4, characterized in that: The FIFO read enable signal is used to control whether the intermediate FIFO buffer unit reads out data, including: When the FIFO read enable signal is 1, the intermediate FIFO buffer unit reads out data; When the FIFO read enable signal is 0, the intermediate FIFO buffer unit does not read data.

6. The intermediate cache scheduling method supporting in-memory CNN according to claim 5, characterized in that: The FIFO write enable signal is used to control the intermediate FIFO buffer unit to write data or output data to the output end of the intermediate buffer scheduling circuit unit, including: When the FIFO write enable signal is 1, the intermediate FIFO buffer unit writes the selection result; When the FIFO write enable signal is 0, the intermediate FIFO buffer unit does not write the selection result, and the write enable signal is inverted to 1 to control data output to the output end of the intermediate buffer scheduling circuit unit.

7. The intermediate cache scheduling method supporting in-memory CNN according to any one of claims 2 to 6, characterized in that: The cache operation logic includes calling each column element in the input array in sequence by row, and obtaining several convolution results corresponding to the element based on the movement relationship between the element and the convolution kernel; among them, each element of the convolution kernel is a weight.

8. The intermediate cache scheduling method supporting in-memory CNN according to claim 7, characterized in that: The convolution operation is performed on the elements of the input array and the convolution kernel weight to obtain the convolution result, which includes calculating the convolution of one element and the weight each time to obtain a convolution result; and storing the result in an intermediate FIFO cache unit through a selection signal and an enable signal.

9. The intermediate cache scheduling method supporting in-memory CNN according to claim 8, characterized in that: When the intermediate cache scheduling method performs a 3*3 convolution kernel operation, the convolution operation on the elements of the input array and the convolution kernel weights to obtain the convolution result further includes: Calculate the convolution of an element in the first row of the input array and the corresponding weight to obtain a convolution result; store the result in the first intermediate FIFO buffer unit through the selection signal and the enable signal; Calculate the convolution of an element in the second row of the input array and the corresponding weight to obtain a convolution result; store the result in the second intermediate FIFO buffer unit through the selection signal and the enable signal; Calculate the convolution of an element in the third row of the input array and the corresponding weight to obtain a convolution result; store the result in the third intermediate FIFO buffer unit through the selection signal and the enable signal; Among them, the convolution of the above three rows of elements and weights is performed simultaneously.

Images