An improved line-buffered sliding window device and its storage method

Abstract

This invention discloses an improved line-buffered sliding window device and its storage method. The device includes a line buffer and a sliding window. The input of the line buffer is connected to a clock signal, a reset signal, input data, an input signal valid signal, a sliding window completion signal, and a line buffer sliding window reset signal. The output of the line buffer is connected to intermediate data, a full signal, a backup buffer full signal, a buffer switching signal, and the line buffer sliding window reset signal. The input of the sliding window is connected to the output of the line buffer, and is also connected to a clock signal, a reset signal, a sliding window enable signal, intermediate data, and a full signal. The output of the sliding window is connected to a sliding window completion signal, a line buffer sliding window reset signal, output data, an output data valid signal, and a sliding window switching signal. This invention adds two signals between the line buffer and the sliding window: a sliding window completion waiting switching signal and a sliding window start signal, enabling the line-buffered sliding window device to adapt to arbitrary convolution strides and fill coefficients.

Description

Technical Field

[0001] This invention relates to a storage optimization device and method for interlayer convolutional neural networks based on in-memory computing devices, belonging to the fields of digital signal processing and photoelectric detection of very large-scale integrated circuits. Background Technology

[0002] Most existing neural network technologies use convolutional deep neural networks, which perform computations on general-purpose computing platforms such as central processing units (CPUs) and graphics processing units (GPUs). Convolutional deep neural networks have large weight parameters and high computational complexity. Traditional in-memory computing systems require extensive access to off-chip caches to read weights when computing convolutional deep neural networks. This not only places high demands on computing power, but also results in significant energy consumption during data transmission, leading to low computational efficiency and failing to fully utilize the actual performance of the computer system.

[0003] In-memory computing devices (IMCs) are computing devices capable of independent computation or integrated with current electronic computing technologies, enabling high-precision in-memory computing functionality. Patent application CN202110804248.3 describes a storage device for convolutional neural network layers based on an IMC, employing a novel linear cache sliding window structure. This structure, during convolutional neural network inference, avoids the traditional operation of storing all feature maps, instead storing only a limited amount of valid data that the IMC array needs to process at the current moment. This significantly reduces the on-chip memory required in actual operation, thereby reducing the chip area used for memory and register flip-flop power consumption. However, this structure is only suitable for convolutional neural networks with a stride of 1, thus limiting its application scenarios. Summary of the Invention

[0004] To optimize hardware area, power consumption, and performance, this invention provides an improved line cache sliding window device and its storage method based on in-memory computing devices, enabling the storage device to adapt to convolutional deep neural networks with different convolution stride requirements and achieve a wider range of application scenarios.

[0005] The technical solution adopted in this invention is as follows:

[0006] An improved line-buffered sliding window device includes a line buffer and a sliding window. The input terminal of the sliding window is connected to the output terminal of the line buffer, and is also connected to a clock signal, a reset signal, a sliding window enable signal, intermediate data, and a full signal. The output terminal of the sliding window is connected to a sliding window completion signal, a line buffer sliding window reset signal, output data, an output data valid signal, and a sliding window switching signal. The sliding window's input terminal also includes a sliding window start signal, and its output terminal includes a sliding window completion waiting-to-switch signal. The input terminal of the line buffer is connected to the clock signal, the reset signal, the input data, the input signal valid signal, the sliding window completion signal, the line buffer sliding window reset signal, and the sliding window completion waiting-to-switch signal. The output terminal of the line buffer is connected to the intermediate data, the full signal, a backup buffer full signal, a buffer switching signal, the line buffer sliding window reset signal, and the sliding window start signal.

[0007] Furthermore, the line cache includes a sliding window cache area and a backup cache area, which are cache arrays arranged together. Data in the backup cache area can be written into the sliding window cache area by shifting the entire array to the left. The size of the backup cache area changes with the convolution stride, so the number of cache columns changes with the convolution stride.

[0008] Furthermore, the sliding window has a built-in counting module, which is used to coordinate the timing between the line buffer and the sliding window when the configurable step size is reached, and finally leads to a new sliding window start signal to indicate that the sliding window starts working.

[0009] This invention also provides a storage method for the above-mentioned line-buffered sliding window device, the specific steps of which include:

[0010] (1) The line buffer includes a sliding window buffer area and a backup buffer area, wherein the storage size of the backup buffer area changes with the convolution stride; the sliding window buffer area stores the data that needs to be slid windowed now, and the backup buffer area stores the data waiting for subsequent sliding windowing; when the sliding window buffer is not full, valid new data is input into the sliding window buffer area;

[0011] (2) After the sliding window buffer area is full, the sliding window buffer full signal inside the line buffer is raised, and valid new data is input into the backup buffer area. When the step size is configurable, there will be cases where the data part of the column to be written to the backup buffer area is zero. Therefore, when identifying the column to be written, it is also necessary to identify the data part of the column as zero, and write the zero and non-zero parts of the data into the backup buffer area respectively. When the sliding window buffer area is full, the line buffer and the sliding window communicate through the sliding window start signal. The sliding window starts to slide and splice the data in the sliding window buffer area, thereby converting the data into the data format required by the subsequent computing layer.

[0012] (3) After the backup cache area is full and the sliding window operation is completed, the data in the backup cache area is transferred to the sliding window cache area to replace part of the data in the sliding window cache area, and a new round of data input is started.

[0013] Compared with the prior art, the advantages of the present invention are:

[0014] (1) The linear buffer sliding window of the present invention only needs to store a number of valid data that the in-memory computing device array needs to process at the current moment according to the size of parameters such as the size of the corresponding convolution kernel of the neural network. Since the amount of intermediate result feature map data to be cached is greatly reduced, the on-chip memory required by its convolutional neural network in actual operation is greatly reduced.

[0015] (2) The original line buffer sliding window device can only be adapted to a convolution stride of 1. The device of the present invention can be adapted to any convolution stride and filling coefficient.

[0016] (3) A sliding window completion waiting switch signal is added to the device of the present invention to keep the sliding window completion signal high, which facilitates the reading of the backup cache column counter. Compared with the original sliding window completion signal, this sliding window completion waiting switch signal is only pulled low when the valid input data signal arrives and the sliding window is completed, thus maintaining the stability and correctness of the line cache sliding window operation.

[0017] (4) The improved line buffer sliding window device of the present invention can adapt to the needs of various convolutional neural networks, and its application scenarios are greatly enriched. It has more obvious advantages over the existing deep neural network in-memory computing architecture. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the in-memory computing architecture used in an embodiment of the present invention. The improved line-buffered sliding window device of the present invention preprocesses the data input to the computing array, corresponding to the preprocessing module in the figure.

[0019] Figure 2 This is a schematic diagram of the online buffered sliding window structure in an embodiment of the present invention. In the diagram, col_done and begin represent the sliding window completion wait signal and sliding window start signal added in the improvement of the present invention.

[0020] Figure 3 This is a schematic diagram of two consecutive feature maps of the convolutional neural network in Example 1. (a) The original feature map sent to the first convolutional layer, and (b) The feature map sent to the second convolutional layer after being processed by the first convolutional layer.

[0021] Figure 4This is a schematic diagram of two consecutive feature maps of the convolutional neural network in Example 2. The left image shows the original feature map fed into the first convolutional layer, where the numbers represent the center point of each sliding window's starting position. The right image shows the feature map fed into the second convolutional layer after processing by the first convolutional layer, where the numbers correspond to the sliding window results in different columns of the left image.

[0022] Figure 5 This is a schematic diagram of the data structure of the line cache section in Example 2. (a) The data structure of the line cache when the second-layer network sliding window is first executed, and (b) The updated data structure of the line cache section. Wherein... Figure 3 Columns with the same data as in (b) also have the same numerical serial numbers. Figure 3 (b) is the same.

[0023] Figure 6 This is a schematic diagram of the feature map of the convolutional neural network in Example 3. Detailed Implementation

[0024] The in-memory computing device in this embodiment uses an opto-in-memory computing unit. The specific structure of its computing unit can use the structure of Embodiments 1, 2, or 3 in patent application CN201910415827.1. The specific structure of the in-memory computing neural network array composed of opto-in-memory computing units can use the structure of Embodiment 4 in patent application CN201910416021.4 or Embodiment 1 in patent application CN202010322172.6, etc. Figure 1 The diagram illustrates the in-memory computing architecture employed in this invention. The computing array is an array composed of multiple in-memory computing device units, storing the weights of the neural network. A high-voltage drive circuit powers the computing array, and a readout circuit collects the results calculated by the computing array, i.e., the current value, and converts them into digital values ​​for transmission to the digital module. Within the digital module, a drive control module controls the high-voltage drive circuit, a preprocessing module preprocesses the data input to the computing array, a data input module provides input to the computing array, and a post-processing module receives, processes, and outputs the data read out by the readout circuit. The improved line buffer sliding window involved in this invention preprocesses the data input to the computing array; this is the preprocessing module shown in the diagram. It can be composed of registers or RAM and is used to distribute data during neural network preprocessing, thereby facilitating in-memory computing in collaboration with the devices.

[0025] A line-buffered sliding window is a storage device between layers of a convolutional neural network based on an in-memory computing device. Its convolutional input structure includes a line buffer and a sliding window. The input of the line buffer is connected to a clock signal, a reset signal, input data, an input signal valid signal, a sliding window completion signal, a line buffer sliding window reset signal, and a sliding window completion waiting switch signal. The output of the line buffer is connected to intermediate data, a full signal, a backup buffer full signal, a buffer switching signal, a line buffer sliding window reset signal, and a sliding window start signal. The input of the sliding window is connected to the output of the line buffer, and also to a clock signal, a reset signal, a sliding window enable signal, intermediate data, and a full signal. The output of the sliding window is connected to a sliding window completion signal, a line buffer sliding window reset signal, output data, an output data valid signal, a sliding window switching signal, and a sliding window completion waiting switch signal. The sliding window start signal and the sliding window completion waiting switch signal are newly added signals in this invention. The original line-buffered sliding window did not have an adjustable stride module, and its timing implementation only supported convolutional sliding windows with a stride of 1. Directly modifying the stride parameter would cause serious timing problems. Analysis revealed that the timing disorder was caused by the original line-buffered sliding window using the sliding window full signal as the start signal. Specifically, when the step size is not 1, using the sliding window full signal as the start signal ignores a step-size-related sliding window wait time, causing the sliding window module to start prematurely, thus resulting in timing errors. To address this, the improved line-buffered sliding window of this invention incorporates a counting module to coordinate the timing between the line-buffered module and the sliding window module when the step size is configurable, ultimately leading to a new sliding window start signal to indicate that the sliding window module has started working. The sliding window start signal ensures that the timing of the line-buffered sliding window remains correct under the configurable step size. Another problem encountered when implementing the configurable step size is that the column count of the backup cache cannot be updated correctly when a column of sliding window completes. Analysis revealed that this problem occurs because the original line-buffered sliding window uses the sliding window completion signal as one of the indication signals for updating the backup cache column count. This sliding window completion signal is pulled high during the clock cycle of sliding window completion and then immediately pulled low. Using this signal as an indicator is feasible when the step size is 1, but when the step size is not 1, the backup cache column counter cannot read the high-level period of this signal in a timely manner. To address this, the improved line cache sliding window of this invention adds a sliding window completion waiting switching signal to keep the sliding window completion signal high, facilitating the reading of the backup cache column counter.

[0026] Example 1

[0027] This embodiment illustrates the improved linear buffer sliding window storage structure and its basic operation in convolutional neural networks. The storage device in this embodiment uses, for example... Figure 2The diagram illustrates a Line Buffer & Sliding Window (LBSW) storage structure. The LBSW performs data reception and transmission functions, processing data input from the upper layer and converting it into the data format required by neural network computation layers such as convolution and pooling. The LBSW consists of a line buffer module (lbsw_bff) and a sliding window module (lbsw_wdw). The line buffer module (lbsw_bff) buffers the data input from the upper layer; the sliding window module (lbsw_wdw) concatenates the buffered data and converts it into the data format required by the lower-level computation layers.

[0028] The input terminals of the line buffer module lbsw_bff are connected to the system clock signal I_sys_clk, the system reset signal I_rst_n, the input data I_data, the input signal valid signal I_data_valid, the line buffer sliding window reset signal rst, and the sliding window completion waiting switch signal I_col_wdw_sw. The output terminals of the line buffer module lbsw_bff are connected to the intermediate data data, the full signal full, the backup buffer full signal O_bak_bff_full, the buffer switch signal O_bff_sw, the line buffer sliding window reset signal O_bw_rst, and the sliding window start signal O_wdw_begin. The input terminals of the sliding window module lbsw_wdw are connected to the system clock signal I_sys_clk, the system reset signal I_rst_n, the sliding window enable signal I_wdw_en, the intermediate data data, the full signal full, and the sliding window start signal I_wdw_begin. The output terminals of the sliding window module lbsw_wdw are connected to the sliding window completion signal done, the line buffer sliding window reset signal rst, the output data O_wdw_data, the output data valid signal O_wdw_data_valid, and the sliding window completion waiting switch signal O_col_wdw_sw.

[0029] The aforementioned Line-Buffered Sliding Window (LBSW) is applied to a convolutional neural network. All weights in this network are arranged in a matrix array, corresponding to the array of opto-in-memory computing (OIM) units, where each unit stores the corresponding weights. The following examples illustrate the implementation details using convolutional neural networks with different original feature map sizes, kernel sizes, padding coefficients, and strides. For different convolutional neural networks, two consecutive convolutional layers are consistently selected for explanation to clarify the working principle of the line-buffered sliding window structure; for multi-channel convolutional neural networks, only one channel is used as an example.

[0030] The basic working principle of the improved line-buffered sliding window of this invention is explained below. Consider the feature map of the classic Alexnet-Cifar10 convolutional neural network, whose original feature map has a single-channel size of 32×32, a single-channel size of the convolutional kernel of 3×3, a convolution stride of 1, and a padding factor of 1. The feature maps of its two consecutive convolutional layers are as follows: Figure 3 As shown. During the first convolutional layer, the sliding window... Figure 3 As shown in the black box at the top left of Figure (a), it slides from the starting position at the top left corner to the bottom left corner with a step size of 1, sequentially generating 32 sets of excitations to supply the first layer array for matrix-vector multiplication calculations and processing, and the result is... Figure 3 The first column region in Figure (b). In the online cache sliding window structure, the result of this column is stored in the first column of the first part of the online cache. After the result of this column is calculated, the sliding window returns to the top left corner and moves 1 unit to the right, with its center moving to the position marked with the number 2 in the second column. Similarly, starting from this position, it slides to the bottom with a step size of 1, and the result is... Figure 3 The region in column 2 of figure (b). In the online cache sliding window structure, the result of this column is stored in column 2 of the first part of the online cache. And so on, after completing the convolution operation and subsequent processing of all feature maps in the first layer, the result is as shown in the figure. Figure 3 Figure (b) shows a new feature map of size 32×32.

[0031] The above-described neural network convolution mapping method illustrates that subsequent convolutional sliding window calculations do not require the complete computation of the new 32×32 feature map before starting; only partial data is needed. Specifically, assuming the single-channel size of the next layer's convolutional kernel remains 3×3, then when... Figure 3 When the data in columns 1, 2, and 3 of Figure (b) are calculated, the second-layer convolutional sliding window operation can already be performed. That is, for a 3×3 convolutional kernel, the array only needs to index three columns of feature map data each time; that is, for each convolutional layer, the first part of the line buffer only needs to store three columns of data. To ensure smooth pipeline flow, the second part of the line buffer needs to cache the amount of data per stride unit as a backup for data transfer between pipeline layers. In the example scenario, the second part of the line buffer needs to cache one column of data as a backup. Specifically, when... Figure 3 In Figure (b), the data in columns 1, 2, and 3 are calculated. When the second layer begins the convolution window operation, the first layer simultaneously calculates the value in column 4 of the right figure and stores it in the second part of the online cache. After the second layer completes the convolution window operation in one column, the three columns of data in the first part of the online cache are updated to columns 2, 3, and 4 for the second layer to perform subsequent convolutions. The second part of the online cache then receives the new window calculation results from the first layer, and so on.

[0032] Example 2

[0033] This embodiment illustrates the basic working principle of the improved linear buffer sliding window of this invention when the convolution stride changes. Consider a convolutional neural network with a single-channel kernel size of 3×3, a single-channel original feature map size of 15×15, a first-layer convolution stride of 2, and a padding factor of 1. The feature maps of its two consecutive convolutional layers are as follows: Figure 4 As shown. The sliding window is made of Figure 4 Starting from the top left corner of the left frame in the left image, it slides to the bottom left corner with a step size of 2, then returns to the right frame position to prepare for another slide. The result of the sliding calculation for the left frame position is as follows. Figure 4 In the right-hand figure, the calculation result in column 1 corresponds to column 2, and so on. Since the convolution stride is 2, the second part of the line buffer needs to buffer 2 columns of data to ensure smooth pipeline flow. The previous line-buffered sliding window used a window buffer full signal O_wdw_bff_full to transmit the start of the sliding window from the line buffer to the sliding window. This worked when the stride was fixed at 1, but caused timing mismatches when the stride was configurable. Therefore, the improved line-buffered sliding window of this invention adds a window start signal O_wdw_begin instead of the window buffer full signal to solve this problem. Figure 5 The image illustrates the data structure of the online cache portion in this embodiment. The three columns on the left represent the sliding window cache, and the two columns on the right represent the backup cache. Each time the sliding window cache is updated, the entire online cache portion is shifted two columns to the left. Figure 5 (a) shows the data structure for the first time the second-layer network sliding window performs line caching. Figure 5 (b) shows the updated line buffer data structure after the second-layer network sliding window completes one column, pulls the window up, and waits for the switching signal O_col_wdw_sw. For ease of understanding, Figure 5 The column number and Figure 4 The column numbers in the right figure correspond one-to-one.

[0034] Example 3

[0035] This embodiment illustrates the processing method of the backup cache of more than one column for zero-value data streams when the improved line-buffered sliding window reaches the feature map boundary and coincides with the filling region, under the condition of configurable convolution stride. Existing line-buffered sliding window devices, due to their non-configurable stride, only need to identify whether to write the entire column of data or fill in the missing zero values ​​when the convolution kernel slides to the boundary region of the feature map. However, the storage method described in this invention differs from existing line-buffered sliding window devices. When identifying the column to be written, it also needs to identify the data portion as a zero column, and write the zero and non-zero data portions into the backup cache respectively. Because the size of the backup cache portion in the improved line-buffered sliding window device of this invention changes with the convolution stride, it ensures that the output sliding window data speed matches the input data speed, thereby realizing the pipelining of the line-buffered sliding window device.

[0036] Assuming the convolution kernel side length is K, the feature map side length is F, the convolution stride is S, and the padding coefficient is P, then the sliding window buffer size is F*K, and the backup buffer size is F*S. Specifically, each feature map layer needs to undergo (F+2*PK) / S+1 top-down sliding window operations. For the backup buffer, in each sliding window operation, the number of times non-zero data needs to be stored is (F+P-K+S-1) / S, defined as bak_sw_cnt. With a configurable stride, in the first bak_sw_cnt-1 operations, complete data columns are stored, while in the bak_sw_cnt-th operation, partially zero columns are stored.

[0037] Consider a convolutional neural network whose original feature map has a single-channel size of 7×7, the convolution kernel also has a single-channel size of 7×7, the convolution stride is 2, and the padding factor is 3. Its feature map is as follows: Figure 6 As shown. This line-buffered sliding window structure requires 7 columns of sliding window buffer and 2 columns of backup buffer to ensure smooth pipeline operation. In this example, the center of the sliding window, i.e., the convolutional kernel, needs to move 3 times horizontally, that is, sliding once on each of the 4 columns marked 1, 2, 3, and 4 in the figure. For aesthetic purposes, Figure 6 The example only shows the starting positions of the convolution kernel when it is located in columns 1 and 4. When the convolution window reaches the boundary, the assignment of the backup buffer needs to be considered. In this example, the value of `bak_sw_cnt` should be 2, meaning that when the center of the convolution kernel is in columns 1 and 2, the backup buffer needs to store non-zero data. The first time, it stores a complete data column, and the second time, it stores a column containing zeros. In the second processing, when the convolution window with the kernel at column 2 finishes processing the corresponding column and switches to a position where the kernel is at column 3, the first column of the backup buffer, `bak_bff1`, will store a column containing zeros, and the second column, `bak_bff2`, will store all zeros. This situation is a new possibility arising from the configurable stride in the improved line-buffered window. This improved line-buffered window adds a register variable `Q_sw` to handle the storage of columns containing zeros. Specifically, in columns where some are 0, only data from rows 0 to Q_sw is stored, while data corresponding to rows with a number greater than Q_sw is set to 0.

Claims

1. An improved line-buffered sliding window device, comprising a line buffer and a sliding window, wherein the input terminal of the sliding window is connected to the output terminal of the line buffer, and is also connected to a clock signal, a reset signal, a sliding window enable signal, intermediate data, and a full signal; the output terminal of the sliding window is connected to a sliding window completion signal, a line buffer sliding window reset signal rst, output data, an output data valid signal, and a sliding window switching signal; characterized in that, The input terminal of the sliding window is provided with a sliding window start signal, and the output terminal of the sliding window is provided with a sliding window completion waiting switch signal; the input terminal of the line buffer is connected to the clock signal, the reset signal, the input data, the input signal valid signal, the sliding window completion signal, the line buffer sliding window reset signal rst, and the sliding window completion waiting switch signal; the output terminal of the line buffer is connected to the intermediate data, the full signal, the backup buffer full signal, the buffer switch signal, the line buffer sliding window reset signal O_bw_rst, and the sliding window start signal; The line cache includes a sliding window cache area and a backup cache area, which are cache arrays arranged together. Data from the backup cache area can be written to the sliding window cache area by shifting the entire array to the left. The size of the backup cache area changes with the convolution stride, so the number of cache columns changes with the convolution stride. The sliding window has a built-in counting module, which is used to coordinate the timing between the line buffer and the sliding window when the configurable step size is reached, and finally leads to a new sliding window start signal to indicate that the sliding window starts working.

2. The storage method using the improved line-buffered sliding window device as described in claim 1, characterized in that, The specific steps include: (1) The line buffer includes a sliding window buffer area and a backup buffer area, wherein the storage size of the backup buffer area changes with the convolution stride; the sliding window buffer area stores the data that needs to be slid windowed now, and the backup buffer area stores the data waiting for subsequent sliding windowing; when the sliding window buffer is not full, valid new data is input into the sliding window buffer area; (2) After the sliding window buffer area is full, the sliding window buffer full signal inside the line buffer is raised, and valid new data is input into the backup buffer area. When the step size is configurable, there will be a situation where the data part to be written to the backup buffer area is a zero column. Therefore, when identifying the column to be written, it is also necessary to identify the data part of the zero column and write the zero and non-zero parts of the data into the backup buffer area respectively. When the sliding window buffer area is full, the line buffer and the sliding window communicate through the sliding window start signal. The sliding window starts to slide and splice the data in the sliding window buffer area, thereby converting the data into the data format required by the subsequent computing layer. (3) After the backup cache area is full and the sliding window operation of the sliding window is completed, the data in the backup cache area is transferred to the sliding window cache area to replace part of the data in the sliding window cache area, and a new round of data input is started.

Images