A multi-layer perceptron device based on a photoelectric storage and computing integrated device

Abstract

The application provides a multi-layer perception device based on a photoelectric storage and calculation integrated device. The device comprises: a general convolution layer module, which is used for performing convolution operation on an input feature map and generating an output feature map transmitted to a mixed layer module; the mixed layer module comprises four mixed layer modules connected in cascade, each mixed layer module comprising a depth separable convolution layer submodule and two fully connected layer submodules connected in sequence; an average pooling and classification output module comprising an average pooling submodule and a classifier submodule, which is used for performing average pooling operation on feature map images output by the four mixed layer modules in channels, and then performing classification output operation on a one-dimensional vector obtained after the average pooling to obtain a final classified prediction result. The neural network accelerator realized by the application has smaller intermediate cache overhead, is more suitable for a calculation array based on a photoelectric storage and calculation integrated device, and improves the efficiency of matrix vector multiplication operation in the neural network.

Description

Technical Field

[0001] This invention relates to a multilayer perceptron device based on an opto-in-memory computing device, belonging to the technical field of hardware-accelerated neural networks. Background Technology

[0002] Driven by the rapid development of the internet industry and the significant improvement in hardware computing power, the wave of artificial intelligence, represented by deep neural networks, has impacted all aspects of human life. Many fields, such as image recognition, autonomous driving, and natural language processing, have been profoundly disrupted by AI technology. However, as Moore's Law approaches its physical limits, and as traditional von Neumann architecture-based general-purpose processors are constrained by memory and power limitations when performing neural network acceleration tasks, AI technology is now facing a dilemma.

[0003] To address these two issues, scientists and researchers have begun to focus on custom hardware acceleration platforms for neural networks. These custom acceleration platforms include improvements based on the traditional von Neumann architecture, as well as new architectures based on in-memory computing devices. Typical in-memory computing devices include memristors and opto-in-memory devices (such as CN110276440A), which can integrate computation and memory access operations, combining the processor and memory into one, resulting in superior performance compared to traditional platforms.

[0004] Opto-in-memory computing devices also have advantages such as non-volatility, low power consumption, and support for large-scale parallel computing. However, current deep neural network algorithms have many layers and complex structures, requiring significant intermediate cache overhead when deployed to in-memory computing platforms for acceleration, and cannot completely avoid off-chip memory access. Summary of the Invention

[0005] In order to establish a neural network accelerator with lower intermediate cache overhead and better adapted to computing arrays based on opto-in-memory computing devices, this invention proposes a multilayer perceptron device based on opto-in-memory computing devices.

[0006] The technical solution adopted in this invention is as follows:

[0007] A multilayer sensor device based on an opto-in-memory computing device includes:

[0008] The ordinary convolutional layer module is used to perform convolution operations on the input feature map and generate an output feature map that is then transmitted to the first hybrid layer module.

[0009] The hybrid layer module includes a cascaded first hybrid layer module, a second hybrid layer module, a third hybrid layer module, and a fourth hybrid layer module. These four hybrid layer modules respectively include a depthwise separable convolutional layer sub-module, a first fully connected layer sub-module, and a second fully connected layer sub-module connected in sequence. They are used to receive the output feature map of the upper-level module and perform a depthwise separable convolution operation and two fully connected operations on the feature map according to the module order. Then, the feature map output by the depthwise separable convolution operation and the output feature map of the second fully connected operation are added point by point and passed to the lower-level module.

[0010] The average pooling and classification output module includes an average pooling submodule and a classifier submodule. It is used to perform average pooling operation on the feature images output by the four mixing layer modules by channel, and then perform classification output operation on the one-dimensional vector obtained after average pooling to obtain the final classification prediction result.

[0011] Furthermore, the ordinary convolutional layer module includes a data storage unit, a parallel-to-serial conversion unit, and a matrix-vector multiplication unit; the data storage unit is used to cache input feature map data and output feature map data; the parallel-to-serial conversion unit is used to expand the input feature map in the data storage unit into a one-dimensional vector according to each channel, and concatenate the first and last ends of the one-dimensional vector to obtain the input activation vector; the matrix-vector multiplication unit is used to perform the matrix-vector multiplication operation mapped from the convolution operation and the fully connected operation.

[0012] Furthermore, the depthwise separable convolutional layer submodule of the first hybrid layer module includes a data storage unit, a parallel-to-serial conversion unit, and a matrix-vector multiplication unit; the depthwise separable convolutional layer submodules of the second, third, and fourth hybrid layer modules include a residual addition and data storage unit, the parallel-to-serial conversion unit, and the matrix-vector multiplication unit; the residual addition and data storage unit is used to add the feature map output by the depthwise separable convolutional layer submodule of the previous hybrid layer module and the feature map output by the second fully connected layer submodule of the previous hybrid layer module point by point.

[0013] Furthermore, the first fully connected layer submodule includes a data register unit, a parallel-to-serial conversion unit, a matrix-to-vector multiplication unit, and a linear rectification unit.

[0014] Furthermore, the second fully connected layer submodule includes a data storage unit, a parallel-to-serial conversion unit, and a matrix-vector multiplication unit.

[0015] Furthermore, the average pooling submodule includes a residual addition and data storage unit and a pooling unit; the residual addition and data storage unit is used to add the feature map output by the depthwise separable convolutional layer submodule of the fourth hybrid layer module and the feature map output by the second fully connected layer submodule of the fourth hybrid layer module point by point.

[0016] Furthermore, the classifier submodule includes a data storage unit, a parallel-to-serial conversion unit, and a matrix-vector multiplication unit.

[0017] Furthermore, the matrix-vector multiplication unit includes a computing array based on an opto-in-memory computing device, a current-to-voltage conversion circuit, an analog-to-digital conversion circuit, and a shift accumulator, used to perform matrix-vector multiplication operations mapped from convolution and fully connected operations.

[0018] Furthermore, the matrix-vector multiplication unit, when implementing the matrix-vector multiplication operation, specifically includes the following steps:

[0019] (1) Convert the M×N weight values ​​into the corresponding photon numbers and store them in a 2 M-row N-column computation array inside the matrix-vector multiplication unit;

[0020] (2) The input feature map of each layer is stored in the data register of that layer. Then the input feature map is expanded into a one-dimensional vector by the parallel-to-serial conversion unit of that layer according to each channel and the first and last parts are concatenated to obtain an M×1 input activation vector.

[0021] (3) Apply the M×1 input excitation vectors to the driving end of each row of the matrix-vector multiplication unit in order from low bit to high bit;

[0022] (4) Finally, the analog current after the current of each column of the matrix-vector multiplication unit is aggregated is converted into a digital result by the current-to-voltage conversion circuit and the analog-to-digital conversion circuit, and then output to the shift accumulator for shift accumulation and summation to obtain the final calculation result.

[0023] Compared with neural network accelerators based on traditional von Neumann architecture chips, this invention has the following advantages:

[0024] (1) This invention proposes a multilayer perceptron device based on opto-in-memory computing devices, and establishes a neural network accelerator with smaller intermediate cache overhead and better adapted to the computing array of opto-in-memory computing devices.

[0025] (2) This invention optimizes the model to address the overfitting phenomenon in the training process of traditional perceptron models by alternately executing depthwise separable convolution and fully connected operations.

[0026] (3) The present invention uses a matrix-vector multiplication unit to perform matrix-vector multiplication calculations after mapping convolution and fully connected operations in the neural network model, which greatly improves the efficiency of matrix-vector multiplication.

[0027] (4) This invention makes full use of the characteristics of opto-in-memory computing devices that can both store and compute, thus reducing the power consumption caused by the repeated access of the computing unit to the off-chip storage and accelerating the inference process of the neural network model. Attached Figure Description

[0028] Figure 1 This is an overall block diagram of the device of the present invention.

[0029] Figure 2 This is a schematic diagram of the internal structure of the device of the present invention.

[0030] Figure 3 This is a schematic diagram of the internal structure of a matrix-vector multiplication unit.

[0031] Figure 4 This is a schematic diagram of the array calculation in the matrix-vector multiplication unit.

[0032] Figure 5 This is a schematic diagram of the optoelectronic in-memory computing device in the embodiment. Detailed Implementation

[0033] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Figure 1 This is an overall block diagram of the multilayer perceptron device based on an opto-in-memory computing device in this embodiment. The network model of the multilayer perceptron includes: a regular convolutional layer module, a hybrid layer 1 module, a hybrid layer 2 module, a hybrid layer 3 module, a hybrid layer 4 module, and an average pooling and classification output module. The regular convolutional layer module is used to perform convolution operations on the input feature map and generate an output feature map that is transmitted to the next-level hybrid layer 1 module. The hybrid layer 1, hybrid layer 2, hybrid layer 3, and hybrid layer 4 modules are connected in sequence to receive the output feature map of the upper-level module and perform a depthwise separable convolution operation and a two-layer fully connected operation on the feature map according to the module order. Then, the feature map output from the depthwise separable convolution operation is added point by point to the feature map output from the second-layer fully connected operation and then transmitted to the next-level module. The average pooling and classification output module is used to perform average pooling operation on the feature image output from the hybrid layer 4 module by channel, and then perform a classification output operation on the one-dimensional vector obtained after average pooling to obtain the final classification prediction result.

[0035] like Figure 2As shown, the ordinary convolutional layer module in this embodiment includes a data storage unit, a parallel-to-serial conversion unit, and a matrix-vector multiplication unit. Each of the four hybrid layer modules includes a depthwise separable convolutional layer submodule, a fully connected layer 1 submodule, and a fully connected layer 2 submodule connected sequentially. Specifically, the depthwise separable convolutional layer submodule of hybrid layer 1 includes a data storage unit, a parallel-to-serial conversion unit, and a matrix-vector multiplication unit; the depthwise separable convolutional layer submodules of hybrid layer 2, hybrid layer 3, and hybrid layer 4 include a residual addition and data storage unit, a parallel-to-serial conversion unit, and a matrix-vector multiplication unit. The fully connected layer 1 submodules of the four hybrid layer modules have the same structure, including a data storage unit, a parallel-to-serial conversion unit, a matrix-vector multiplication unit, and a linear rectification unit. The fully connected layer 2 submodules of the four hybrid layer modules also have the same structure, including a data storage unit, a parallel-to-serial conversion unit, and a matrix-vector multiplication unit.

[0036] Among the above units, the data storage unit is used to cache the input feature map data and output feature map data of each module or sub-module; the parallel-to-serial conversion unit is used to expand the input feature map in the data storage unit into a one-dimensional vector according to each channel, and concatenate the first and last ends of the one-dimensional vector to obtain the input activation vector; the matrix-vector multiplication unit is used to perform the matrix-vector multiplication operation after the convolution operation and the fully connected operation mapping; and the linear rectification unit is used to add the ReLU (linear rectification) activation function operation in the fully connected layer 1 sub-module.

[0037] The average pooling and classification output module includes an average pooling submodule and a classifier module connected in sequence. The average pooling submodule includes a residual addition and data storage unit, as well as a pooling unit. The classifier submodule includes a data storage unit, a parallel-to-serial conversion unit, and a matrix-vector multiplication unit. The residual addition and data storage unit is used to add the feature maps output by the depthwise separable convolutional layer submodule and the feature maps output by the fully connected layer 2 submodule point by point. The pooling unit is used to perform average pooling on the feature images by channel.

[0038] Figure 3 This is a schematic diagram of the matrix-vector multiplication unit in this embodiment. The matrix-vector multiplication unit includes a computing array based on an opto-in-memory computing device, a current-to-voltage conversion circuit, an analog-to-digital conversion circuit, and a shift accumulator, used to perform matrix-vector multiplication operations after convolution and fully connected operations. Because the number of weights required by each of the above modules to perform matrix-vector multiplication operations differs, the computing arrays contained in the matrix-vector multiplication units of each module are also different in size.

[0039] Figure 4 This is a schematic diagram of the computing array in this embodiment. The computing array is an M×N cross array composed of opto-in-memory computing devices. Each row of the computing array is called a word line, and each column is called a bit line. The opto-in-memory computing devices are located at the intersection of rows and columns.

[0040] Figure 5 This is a schematic diagram of the opto-in-memory computing device in this embodiment. A single device includes a gate, an optical input terminal, a source, a substrate, and a drain. The source is the output terminal for the computation results, and the gate and drain together serve as the control terminals to control the switching of the device. The opto-in-memory computing device can store the number of photons received at the optical input terminal within the device.

[0041] The opto-in-memory computing devices in the aforementioned computing array can only store non-negative numbers, but the weights required for convolution and fully connected operations in neural networks may be negative. Therefore, the computing array uses a double array method to store weights: the first array stores positive weights, the second array stores the absolute values ​​of negative weights, and the remaining positions in both arrays are filled with 0; then, during matrix-vector multiplication, the result of the operation in the first array is subtracted from the result of the operation in the second array to obtain the complete result including negative weights.

[0042] according to Figure 3 and Figure 4 In this embodiment, the matrix-vector multiplication unit performs the matrix-vector multiplication operation by including the following steps:

[0043] (1) First, the M×N weight values ​​are converted into the corresponding photon numbers and stored in two M-row N-column computation arrays inside the matrix-vector multiplication unit.

[0044] (2) Next, the input feature map of each layer is stored in the data register unit of that layer. Then, the input feature map is expanded into a one-dimensional vector by the parallel-to-serial conversion unit of that layer according to each channel and the first and last parts are concatenated to obtain an M×1 input excitation vector.

[0045] (3) Apply the M×1 input excitation vectors to the driving end of each row of the matrix-vector multiplication unit in order from low bit to high bit;

[0046] (4) Finally, the analog current after the current of each column of the matrix-vector multiplication unit is aggregated is converted into a digital result by the current-to-voltage conversion circuit and the analog-to-digital conversion circuit, and then output to the shift accumulator for shift accumulation and summation to obtain the final calculation result.

[0047] Based on the principle of matrix-vector multiplication using the above-mentioned computational array, after mapping the convolution and fully connected operations of each layer in this embodiment to matrix-vector multiplication operations, the computational array size of the matrix-vector multiplication unit of each module is shown in Table 1.

[0048] Table 1 shows the calculation array size of the matrix-vector multiplication unit for each module in the embodiment.

[0049]

[0050] Based on this, the calculation process of this embodiment is as follows:

[0051] (1) The ordinary convolutional layer module performs a convolution operation with a kernel size of 4×4×3×128 and a stride of 4 on the input feature map with a size of 32×32×3. First, the weights of 128 4×4×3 convolutional kernels, quantized to 6 bits, are stored in the computation array of the matrix-vector multiplication unit (since the computation array divides the weights into positive and negative columns for separate operations, the number of columns in the array is twice the number of convolutional kernels, so the weight array is 48 rows × 256 columns). Meanwhile, an input feature map, quantized to 8 bits and measuring 32×32×3×8 bits, is stored in the data register unit. Next, the 4×4×3×8-bit feature map blocks in the data register unit are converted into 48×8-bit activation data through a parallel-to-serial conversion unit and enter the matrix-vector multiplication unit to perform a matrix-vector multiplication operation with each of the 128 convolutional kernel weights in the computation array. The data is then stored in the data register unit of the depthwise separable convolutional layer submodule of the hybrid layer 1 module. After the ordinary convolutional layer module performs 64 matrix-vector multiplication operations, an 8×8×128×8-bit feature map is generated in the data register unit of the depthwise separable convolutional layer submodule of the hybrid layer 1 module.

[0052] (2) The Hybrid Layer 1 module first performs a depthwise separable convolution operation on the 8×8×128×8 bit feature map data in the data register unit with a kernel size of 3×3×1×128 and a stride of 1. This is accomplished by the depthwise separable convolution layer submodule. The depthwise separable convolution layer submodule first stores the weight data of 128 3×3×1 convolution kernels quantized to 6 bits in the array of its matrix-vector multiplication units (since the depthwise separable convolution performs convolution operation on each channel of the input feature map separately, the weight array is 9 rows × 2 columns × 128), and zero-padding the feature image in the data register unit to 10×10×128×8 bits; next, the 3×3×8 bit feature map block data in the data register unit is converted to 9 bits by the parallel-to-serial conversion unit. The 8-bit stimulus enters the matrix-vector multiplication unit and performs a matrix-vector multiplication operation with each of the 128 convolutional kernel weights in the array. Following the fusion principle, this matrix-vector multiplication operation, after being registered, can be directly applied to the fully connected layer 1 submodule. The fully connected layer 1 submodule is responsible for expanding the image dimension to 64×512, and its input stimulus is 64×128×8 bits. Therefore, the fully connected operation it performs has 128×512 weight values ​​(hence the weight array is 128 rows × 1024 columns). After performing a matrix-vector multiplication operation, the matrix-vector multiplication unit of the Layer 1 submodule adds a ReLU (Rectified Luminaire) activation function to the output data and outputs the data to the fully connected Layer 2 submodule. The fully connected Layer 2 submodule is responsible for reducing the image dimension to 64×128, and its input stimulus is 64×512×8 bits. Therefore, the fully connected operation it performs has 512×128 weight values ​​(hence the weight array is 512 rows × 256 columns). The fully connected Layer 1 submodule... The input stimulus is 64×128×8 bits, and the fully connected operation it performs has 128×512 weight values ​​(therefore, the weight array is 512 rows × 1024 columns). After performing the matrix-vector multiplication operation in the fully connected layer 2 submodule, the output result is input together with the output result of the depthwise separable layer submodule into the residual summation and data storage unit of the depthwise separable layer submodule in the hybrid layer 2 module. At this time, the input feature map size of the hybrid layer 2 module is still 8×8×128×8 bits. The hybrid layer 2, hybrid layer 3, and hybrid layer 4 modules each contain one depthwise separable convolutional layer submodule and two fully connected layer submodules. Their operating mode is the same as that of the hybrid layer 1 module, except that the input of the depthwise separable layer submodule is different; the rest of the operations are completely identical, so they are omitted.

[0053] (3) The average pooling and classification output module includes an average pooling submodule and a classifier submodule. In this embodiment, the average pooling submodule is implemented using a 6-level digital logic adder tree, with registers added between each adder level. The average pooling submodule is used to perform an 8×8 average pooling operation on the input feature map of size 8×8×128, that is, to average the 8×8 data matrix in each channel, which will result in a 1×128 one-dimensional vector. The final classifier submodule is used to transform the dimension of the pooled 1×128 one-dimensional vector to 1×10 to obtain the final classification prediction result. This is also implemented through a fully connected operation using a matrix-vector multiplication unit. Its input excitation is 1×128×8 bits, so the fully connected operation it performs has 128×10 weight values ​​(therefore, the weight array is 128 rows × 20 columns).

[0054] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A multilayer sensor device based on an optoelectronic in-memory computing device, characterized in that, The multilayer sensing device includes: The ordinary convolutional layer module is used to perform convolution operations on the input feature map and generate an output feature map that is then transmitted to the first hybrid layer module. The hybrid layer module includes a cascaded first hybrid layer module, a second hybrid layer module, a third hybrid layer module, and a fourth hybrid layer module. These four hybrid layer modules respectively include a depthwise separable convolutional layer sub-module, a first fully connected layer sub-module, and a second fully connected layer sub-module connected in sequence. They are used to receive the output feature map of the upper-level module and perform a depthwise separable convolution operation and two fully connected operations on the feature map according to the module order. Then, the feature map output by the depthwise separable convolution operation and the output feature map of the second fully connected operation are added point by point and passed to the lower-level module. The average pooling and classification output module includes an average pooling submodule and a classifier submodule. It is used to perform average pooling on the feature images output by the four mixing layer modules by channel, and then perform classification output on the one-dimensional vector obtained after average pooling to obtain the final classification prediction result. The ordinary convolutional layer module includes a data storage unit, a parallel-to-serial conversion unit, and a matrix-vector multiplication unit. The data storage unit is used to cache input feature map data and output feature map data. The parallel-to-serial conversion unit is used to expand the input feature map in the data storage unit into a one-dimensional vector according to each channel, and concatenate the first and last ends of the one-dimensional vector to obtain the input activation vector. The matrix-vector multiplication unit is used to perform matrix-vector multiplication operations after convolution and fully connected operations, and it includes a computing array based on optoelectronic in-memory computing devices, a current-to-voltage conversion circuit, an analog-to-digital conversion circuit, and a shift accumulator. The matrix-vector multiplication unit, when implementing the matrix-vector multiplication operation, specifically includes the following steps: (1) Convert the M×N weight values ​​into the corresponding photon numbers and store them in two M-row N-column computation arrays inside the matrix-vector multiplication unit; the first array stores the positive weights, the second array stores the absolute values ​​of the negative weights, and the remaining positions of the two arrays are filled with 0; (2) The input feature map of each layer is stored in the data register of that layer. Then the input feature map is expanded into a one-dimensional vector by the parallel-to-serial conversion unit of that layer according to each channel and the first and last parts are concatenated to obtain an M×1 input activation vector. (3) Apply M×1 input excitation vectors to the driving end of each row of the calculation array inside the matrix-vector multiplication unit in order from low bit to high bit; when the matrix-vector multiplication unit is calculated, subtract the calculation result of the second array from the calculation result of the first array to obtain the complete calculation result containing negative weights. (4) Finally, the analog current after the current of each column of the matrix-vector multiplication unit is aggregated is converted into a digital result by the current-to-voltage conversion circuit and the analog-to-digital conversion circuit, and then output to the shift accumulator for shift accumulation and summation to obtain the final calculation result.

2. The multilayer sensor device based on an optoelectronic in-memory computing device according to claim 1, characterized in that, The depthwise separable convolutional layer submodule of the first hybrid layer module includes a data storage unit, a parallel-to-serial conversion unit, and a matrix-vector multiplication unit; the depthwise separable convolutional layer submodules of the second, third, and fourth hybrid layer modules include a residual addition and data storage unit, the parallel-to-serial conversion unit, and the matrix-vector multiplication unit; the residual addition and data storage unit is used to add the feature map output by the depthwise separable convolutional layer submodule of the previous hybrid layer module and the feature map output by the second fully connected layer submodule of the previous hybrid layer module point by point.

3. The multilayer sensor device based on an optoelectronic in-memory computing device according to claim 1, characterized in that, The first fully connected layer submodule includes a data register unit, a parallel-to-serial conversion unit, a matrix-to-vector multiplication unit, and a linear rectification unit.

4. A multilayer sensor device based on an optoelectronic in-memory computing device according to claim 1, characterized in that, The second fully connected layer submodule includes a data storage unit, a parallel-to-serial conversion unit, and a matrix-vector multiplication unit.

5. A multilayer sensor device based on an optoelectronic in-memory computing device according to claim 1, characterized in that, The average pooling submodule includes a residual addition and data register unit and a pooling unit; the residual addition and data register unit is used to add the feature map output by the depthwise separable convolutional layer submodule of the fourth hybrid layer module and the feature map output by the second fully connected layer submodule of the fourth hybrid layer module point by point.

6. A multilayer sensor device based on an optoelectronic in-memory computing device according to claim 1, characterized in that, The classifier submodule includes a data storage unit, a parallel-to-serial conversion unit, and a matrix-vector multiplication unit.

Images