High-throughput convolutional neural network recursive weight mapping method of storage and calculation integrated architecture

Abstract

The invention relates to a recursive weight mapping method for a high-throughput convolutional neural network based on a storage and calculation integrated architecture, and the method comprises the steps: forming a basic unit of a storage and calculation integrated chip based on a two-dimensional array structure of a nonvolatile memory, and storing the weight information in the convolutional neural network; comprising a single-layer CNN network-oriented weight replication mapping method and a multi-layer CNN network-oriented recursive weight replication mapping method, and the single-layer CNN network-oriented weight replication mapping method comprises the following steps: mapping a CNN model into a plurality of nonvolatile memory arrays by using a mapping tool, determining a weight replication coefficient, and performing recursive weight replication mapping on the plurality of nonvolatile memory arrays; through a method based on an expansion window, the weight is mapped into a nonvolatile memory array so as to achieve a weight copy mapping method. On the basis that multiple layers are single layers, the weight copy coefficient of the next layer is determined by judging whether an interlayer pooling operator exists or not, and the process is iterated until the last layer of the network.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a high-throughput convolutional neural network recursive weight mapping method based on a storage-computing integrated architecture. Background Art

[0002] In recent years, artificial intelligence has developed rapidly. Traditional AI models based on convolutional neural networks (CNN) still have unshakable application value in most cost- and resource-constrained scenarios (such as drone and vehicle-mounted target recognition) due to their advantages of small number of parameters and computational complexity. CNN is a deep learning model specifically designed to process data with a grid structure. CNN simulates the biological visual system through a combination of convolutional layers, pooling layers, and fully connected layers, uses convolution operations to extract local features, then reduces the size of feature maps through pooling layers, and finally performs classification through fully connected layers. It performs well in processing image and video data, can effectively extract image features, and is applied to tasks such as image classification and object detection. It has become a mainstream method in the field of computer vision.

[0003] The time overhead of CNN specifically occurs in operations related to pooling. The pooling operation in the existing CNN is to perform an operation on a certain area of ​​the output of its predecessor operation. The pooling operation depends on multiple data of the predecessor operation. However, these data must be obtained through several window sliding within a certain period of time. When the convolution computing hardware performance and communication network performance are large enough, convolution window sliding is a key factor limiting the performance of existing convolutional networks.

[0004] Therefore, it is necessary to provide a solution in which the memory is given the function of computing in the storage-computing integrated architecture, so that the data generated by the memory can be calculated on the spot without the need to move back and forth between the memory and the processor, thereby significantly reducing data movement and eliminating memory constraints. Among the existing storage-computing integrated implementations, the storage-computing integrated based on non-volatile memory has the characteristics of high density, fast read access and low leakage power consumption. It can efficiently combine the CNN algorithm and form a storage-computing integrated intelligent computing architecture with large-scale matrix computing capabilities. Therefore, it is considered to be an ideal choice for the future deployment of artificial intelligence algorithms.

[0005] Existing non-volatile memory devices are generally implemented in a two-dimensional cross structure, in which the horizontal wires are called word lines and the vertical wires are called bit lines. Each word line is connected to each bit line through a resistive memory cell. By writing a certain resistance value in the non-volatile memory in advance and applying different voltages to different bit lines, the sum of the currents stimulated by all memory cells in the current column can be obtained on the bit line according to Kirchhoff's law. This shows that the calculation mode of a bit line in the non-volatile memory is a vector inner product operation. When combined together in a two-dimensional array of non-volatile memory, matrix-vector multiplication is obtained.

[0006] Deploying in-memory computing core units based on the mesh topology on-chip network can significantly reduce communication latency. Under a certain weight mapping framework, the classic CNN model is mapped to the integrated computing unit (Tile). Based on the computing and routing characteristics of the Tile, the on-chip network communication structure is planned, and finally the overall mapping of the CNN model to the integrated computing architecture is realized. The data flow of the integrated computing architecture determines the communication structure, so that the optimization of network performance can be transferred to data computing. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a high-throughput convolutional neural network recursive weight mapping method based on a storage-computing integrated architecture. The mapping method is based on a two-dimensional array structure of a non-volatile memory to form a basic unit of a storage-computing integrated chip, which stores weight information in the convolutional neural network, including a weight copy mapping method for a single-layer CNN network and a recursive weight copy mapping method for a multi-layer CNN network. The environment construction part considers a simple convolution layer, which only involves two operations, convolution and pooling. The convolution kernel size is 3×3, the convolution step size is 1, and the input channel and output channel are both 1. Pooling operates on each 2×2 area of ​​the feature map output by the previous operation. The weight copy mapping method for a single-layer CNN network has the following steps:

[0008] Step S1: first use a mapping tool to map the CNN model to multiple non-volatile memory arrays, and then consider the convolution operation of a certain convolution layer in the CNN;

[0009] Step S2: Determine a weight replication coefficient. The width of the input feature map of the convolution layer is 11, the height is large enough, the input channel Ci=3, the output channel Co=2, the convolution window size is 3×3, the step size is 1, and the weight replication coefficient N is 4.

[0010] Step S3: The weights are mapped to the non-volatile memory array by a method based on an extended window; the extended window size of the convolution layer is 3×6, the window horizontal sliding step is 4, and the line step remains unchanged; at time T0, the 3×6 convolution window moves to the starting position of the input feature map, and the contained data is arranged in sequence to the input port of Xbar in a row-first order, and the weights are filled into the non-volatile memory inside Xbar according to the corresponding position of the input data, and the free position is filled with 0, waiting for the ADC to output the calculated value, which is the 4 data on the two output channels of the operation output feature map; and as the extended window slides, the Xbar input port data changes, and the next 4 data on the two output channels of the operation output feature map are output under the same weight arrangement;

[0011] After completing the above steps, it is determined whether the next layer contains a pooling operator, and the weight replication coefficient N is adjusted accordingly, so that the weight replication operation of the next layer can iteratively adapt to the throughput improvement of the previous layer, so as to achieve the effect of multi-layer weight replication to ensure the overall throughput between layers, that is, a recursive weight replication mapping method for multi-layer CNN networks.

[0012] In one embodiment of the present invention, after the mapping tool in step S1 maps the CNN model to multiple non-volatile memory arrays, a weight replication coefficient needs to be determined, and the extended window size is closely related to the weight replication multiple, which determines how many ordinary convolution window sizes the extended window covers at one time.

[0013] In one embodiment of the present invention, when the weight replication coefficient N=4 in step S2, the 3×6 extended convolution window is N equivalent convolution windows with a step size of 1 and a size of 3×3; if a convolution layer with a window size of 7×7 and a step size of 2 requires N=16 times weight replication, its equivalent window size is 7×37 (7+2*(16-1)=37).

[0014] In one embodiment of the present invention, after step S2 performs weight replication on the current convolutional layer, since the next convolutional layer does not perform corresponding weight replication, in order to improve the data processing capability of the next convolutional layer, it is necessary to perform recursive weight replication on the convolutional network.

[0015] In one embodiment of the present invention, changing the shape and size of the convolution window in step S3 does not change the convolution operation itself, but only increases the single data flow by expanding the convolution window.

[0016] In one embodiment of the present invention, the convolution layer used in the recursive weight copying method in step S3 involves only two operations: convolution and pooling. The convolution kernel size is 3×3, the convolution step is 1, the input channel and the output channel are both 1, and the pooling operates on each 2×2 area of ​​the feature map output by the previous operation.

[0017] In one embodiment of the present invention, the recursive weight replication method in step S3 reduces the communication delay by 41.71% in a simple 3-layer convolutional neural network without consuming additional resources.

[0018] In one embodiment of the present invention, the use of an expanded 3×6 convolution window in step S3 increases the throughput by 4 times compared to a 3×3 convolution window.

[0019] The above technical solution of the present invention has the following advantages over the prior art: the high-throughput convolutional neural network recursive weight mapping method described in the present invention pre-lays out the mapping of weights in the non-volatile memory array, which can improve the throughput in a single-layer convolution layer. Based on this method, this patent further proposes an iterative weight replication mapping scheme, which can maintain the high throughput characteristics of a single layer in a multi-layer CNN network; at the same time, it solves the important time bottleneck in CNN reasoning, improves the inter-layer throughput, and when other factors are not affected, uses an extended 3×6 convolution window to increase the throughput by 4 times compared to a 3×3 convolution window. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0021] Figure 1 1 is a flow chart of the input-output dependency of a simple convolutional layer using a 3×3 convolutional window in the prior art, wherein (a) the convolution operation and pooling operation of the 3×3 kernel window; (b) the input data and output data of such a convolutional layer in the time dimension;

[0022] Figure 2 Schematic diagram of the input-output dependency of a simple convolutional layer using a 3×6 extended convolutional window in the present invention, wherein (a) the convolution operation and pooling operation of the 3×6 kernel window; (b) the input data and output data of such a convolutional layer in the time dimension;

[0023] Figure 3 is a flow chart of the recursive weight replication of the present invention, wherein (a) reverse reasoning is performed in the typical CNN calculation process; (b) the weight replication coefficient N of each convolutional layer is recursively calculated;

[0024] Figure 4It is a flowchart of weight replication in the CIM architecture described in the present invention, and specifically, the input channel of the convolution layer is 3, the output channel is 2, (a), the expanded window coverage on the input feature map; (b), the original convolution kernel and weights; (c), from T0 to Tn, the correspondence between the input data and the weights in Xbar, and the output data of the convolution operation; (d), the feature map after the convolution operation. DETAILED DESCRIPTION

[0025] like Figure 1 The input-output dependency of the process is shown in the figure. In the prior art, a simple convolution layer is considered. The convolution layer only involves two operations: convolution and pooling. The convolution kernel size is 3×3, the convolution step size is 1, and the input channel and output channel are both 1. Pooling operates on each 2×2 area of ​​the feature map output by the previous operation. The previous convolution layer of this convolution layer sends a feature map of size 4×4. After padding, the input feature map size becomes 6×6. According to the data flow characteristics of the weight mapping, after 6 convolution window slides, the convolution window slides to the second pixel of the second row of the input feature map. The pooling layer obtains complete data once and outputs the first pixel of the output feature map. After every 2 convolution window slides, the pooling layer outputs one pixel until the sliding window slides to the end of the row of the input feature map. After the sliding window slides 6 times again, the pooling layer starts to output one pixel every 2 convolution window slides again.

[0026] like Figure 2 As shown, this embodiment provides a high-throughput convolutional neural network recursive weight mapping method based on a storage-computing integrated architecture. The mapping method is based on a two-dimensional array structure of a non-volatile memory to form a basic unit of a storage-computing integrated chip, which stores weight information in the convolutional neural network internally, making full use of the characteristics of the storage-computing integrated architecture, so that for the same input feature map, the number of window sliding times can be reduced while improving the pooling output throughput; including:

[0027] Still consider a simple convolution layer, which only involves two operations: convolution and pooling. The convolution kernel size is 3×3, the convolution step is 1, the input channel and the output channel are both 1, and the pooling operates on each 2×2 area of ​​the feature map output by the previous operation. In this embodiment, a 3×6 convolution window is set for the convolution layer instead of 3×3. The horizontal sliding step of the window is changed to 4, and the line sliding step remains unchanged at 1. It is worth noting that changing the shape and size of the convolution window does not change the convolution operation itself, but only increases the single data flow by expanding the convolution window. By Figure 2It can be seen that after increasing the convolution window, while the number of overall window sliding times is reduced, the output throughput of the convolution layer has been significantly improved. The data output from the pooling operation forms a complete data stream, which can effectively solve the rough estimation problem. When other factors are not affected, using an extended 3×6 convolution window can increase the throughput by 4 times compared to a 3×3 convolution window.

[0028] The extended window size is closely related to the weight replication factor, which determines how many normal convolution window sizes the extended window can cover at one time. In this example, with a weight replication factor of N = 4, a 3×6 extended convolution window is an equivalent convolution window of N steps of 1 and size 3×3. If a convolution layer with a window size of 7×7 and a stride of 2 requires N = 16 weight replications, its equivalent window size is 7×37 (7+2*(16-1) = 37).

[0029] In a simple convolutional network, the weights of the first convolutional layer are replicated in the above-mentioned extended window, and the throughput of the first layer is theoretically increased by four times. However, since the second convolutional layer does not perform corresponding weight replication, the data sent by the first layer to the second layer exceeds the processing capacity of the second layer, and the convolution calculation of the second layer becomes the new bottleneck of the convolutional network. Therefore, recursive weight replication should be performed on the convolutional network.

[0030] The schematic diagram of the recursive weight replication scheme is as follows Figure 3 As shown in (a). Starting from a middle layer of the convolutional network, reverse reasoning is performed. After a pooling operation, the weight of the convolutional layer where the pooling operation is located is copied 4 times. The convolutional layer that has not been pooled retains the weight copy multiple of its subsequent convolutional layer until the first convolutional layer is derived. The above recursive idea can be used Figure 3 The flowchart in (b) is shown.

[0031] Specifically, the high-throughput convolutional neural network recursive weight mapping method described in this embodiment is to first use existing mapping tools to configure each convolutional layer in the CNN into a non-volatile memory array in the storage and computing integrated architecture.

[0032] Figure 4 The above considers the convolution operation of a convolution layer in CNN. The width of the input feature map of this layer is 11, the height is large enough, the input channel Ci=3, the output channel Co=2, the convolution window size is 3×3, the step size is 1, and the weight replication coefficient is 4. According to the conclusion of the previous section, the expansion window size of this layer is 3×6, the window horizontal sliding step size is 4, and the line break step size remains unchanged. Indicates the sixth value of the second channel of the input feature map, and C1 represents a weight value on the second input channel of a convolution kernel. In the process of representing data transfer to Tile, use Represents an input tensor consisting of the values ​​of three input channels at the same position, and uses C={C0,C1,C2} to represent a weight tensor consisting of the corresponding weights of the three input channels at the same position.

[0033] At time T0, the 3×6 convolution window moves to the starting position of the input feature map, and arranges the contained data in row-first order to the input port of Xbar. The weights are filled into the non-volatile memory inside Xbar according to the corresponding position of the input data, and the empty positions are filled with 0, waiting for the ADC to output the calculated value, which is the 4 data on the two output channels of the output feature map of this operation. As the expansion window slides, the data of the Xbar input port changes, and the next 4 data on the two output channels of the output feature map of this operation are output under the same weight arrangement.

[0034] After completing the weight copy operation of a single layer, you can use Figure 3 The flowchart in (b) determines whether the next layer contains a pooling operator, and adjusts the weight replication coefficient N accordingly, so that the weight replication operation of the next layer can iteratively adapt to the throughput improvement of the previous layer, so as to achieve the effect of multi-layer weight replication to ensure the overall throughput between layers.

[0035] In summary, the mapping method described in this embodiment uses this method to recursively copy and map the CNN weights under the storage-computing integrated architecture, which can solve the important time bottleneck in CNN reasoning and improve the inter-layer throughput. When other factors are not affected, using an extended 3×6 convolution window increases the throughput by 4 times compared to a 3×3 convolution window.

[0036] At the same time, the overall hardware communication delay will be greatly reduced under the condition of acceptable increase in hardware resource consumption. In a simple 3-layer convolutional neural network, the use of recursive weight replication can reduce the communication delay by 41.71% without additional resource consumption.

[0037] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A high-throughput convolutional neural network recursive weight mapping method based on a storage-computing integrated architecture, wherein the mapping method is based on a two-dimensional array structure of a non-volatile memory to form a basic unit of a storage-computing integrated chip, which stores weight information in a convolutional neural network, and is characterized in that: It includes a weight copy mapping method for a single-layer CNN network and a recursive weight copy mapping method for a multi-layer CNN network. The environment construction part considers a simple convolution layer, which only involves two operations: convolution and pooling. The convolution kernel size is 3×3, the convolution step is 1, and the input channel and output channel are both 1. Pooling operates on each 2×2 area of ​​the feature map output by the previous operation. The weight copy mapping method for a single-layer CNN network is as follows: Step S1: first use a mapping tool to map the CNN model to multiple non-volatile memory arrays, and then consider the convolution operation of a certain convolution layer in the CNN; Step S2: Determine a weight replication coefficient. The width of the input feature map of the convolution layer is 11, the height is large enough, the input channel Ci=3, the output channel Co=2, the convolution window size is 3×3, the step size is 1, and the weight replication coefficient N is 4. Step S3: The weights are mapped to the non-volatile memory array by an extended window-based method; the extended window size of the convolution layer is 3×6, the horizontal sliding step of the window is 4, and the line step remains unchanged; at time T0, the 3×6 convolution window moves to the starting position of the input feature map, and the contained data is arranged in sequence to the input port of Xbar in row-first order, and the weights are filled into the non-volatile memory inside Xbar according to the corresponding position of the input data, and the free position is filled with 0, waiting for the ADC to output the calculated value, which is the 4 data on the two output channels of the output feature map of the operation; As the extended window slides, the Xbar input port data changes, and the next 4 data on the two output channels of the output feature map of the operation are output under the same weight arrangement; After completing the above steps, it is determined whether the next layer contains a pooling operator, and the weight replication coefficient N is adjusted accordingly, so that the weight replication operation of the next layer can iteratively adapt to the throughput improvement of the previous layer, so as to achieve the effect of multi-layer weight replication to ensure the overall throughput between layers, that is, a recursive weight replication mapping method for multi-layer CNN networks.

2. The high-throughput convolutional neural network recursive weight mapping method according to claim 1, characterized in that: After the mapping tool in step S1 maps the CNN model to multiple non-volatile memory arrays, a weight replication coefficient needs to be determined, and the extended window size is closely related to the weight replication multiple, which determines how many ordinary convolution window sizes the extended window covers at one time.

3. The high-throughput convolutional neural network recursive weight mapping method according to claim 1, characterized in that: When the weight replication coefficient N=4 in step S2, the 3×6 extended convolution window is an equivalent convolution window of N with a step size of 1 and a size of 3×3; if a convolution layer with a window size of 7×7 and a step size of 2 requires N=16 times the weight replication, its equivalent window size is 7×37.

4. The high-throughput convolutional neural network recursive weight mapping method according to claim 1, characterized in that: After step S2 copies the weights of the current convolutional layer, since the next convolutional layer does not perform corresponding weight copying, in order to improve the data processing capability of the next convolutional layer, the convolutional network needs to be recursively copied.

5. The high-throughput convolutional neural network recursive weight mapping method according to claim 1, characterized in that: Changing the shape and size of the convolution window in step S3 does not change the convolution operation itself, but only increases the single data flow by expanding the convolution window.

6. The high-throughput convolutional neural network recursive weight mapping method according to claim 1, characterized in that: The convolution layer used in the recursive weight replication method in step S3 involves only two operations: convolution and pooling. The convolution kernel size is 3×3, the convolution step is 1, the input channel and the output channel are both 1, and the pooling operates on each 2×2 area of ​​the feature map output by the previous operation.

7. The high-throughput convolutional neural network recursive weight mapping method according to claim 1, characterized in that: The recursive weight replication method in step S3 reduces the communication delay by 41.71% in a simple 3-layer convolutional neural network without consuming any additional resources.

8. The high-throughput convolutional neural network recursive weight mapping method according to claim 1, characterized in that: The use of an expanded 3×6 convolution window in step S3 increases the throughput by 4 times compared to a 3×3 convolution window.

Images