Optical tensor calculation accelerator based on multi-imaging projection and calculation method thereof
By using an optical tensor computing accelerator based on multi-imaging projection and leveraging optical parallelism and multiplexing techniques, the resource consumption problem of tensor convolution computation in electronic computing is solved, achieving low-latency and high-efficiency tensor convolution operations.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI INST OF OPTICS & FINE MECHANICS CHINESE ACAD OF SCI
- Filing Date
- 2022-08-24
- Publication Date
- 2026-04-17
AI Technical Summary
Existing electronic computers suffer from excessive resource consumption and storage requirements in tensor convolution calculations, especially in artificial neural networks, where traditional methods require additional storage resources and energy consumption that is difficult to reduce.
An optical tensor computation accelerator based on multi-imaging projection is employed. By rearranging the input feature map and convolution kernel matrix, wavelength division multiplexing and polarization multiplexing are used to realize the output of multi-channel convolution kernels, and tensor computation is performed by combining optical parallel characteristics.
It achieves efficient, low-latency, and low-power tensor convolution computation, reducing the consumption of additional storage resources and making it suitable for efficient computation of optical neural networks.
Smart Images
Figure CN115392446B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of optical computing, specifically to an optical tensor computing accelerator based on multiple imaging projections and its computing method. Background Technology
[0002] The rapid development of artificial intelligence (AI) technologies, exemplified by deep neural networks, marks humanity's entry into the era of big data. In today's information explosion, the growth rate of massive data computing demands far exceeds the supply provided by Moore's Law. Computing speed and power consumption are increasingly becoming hardware bottlenecks for AI development. Faced with the computing dilemmas of the post-Moore's Law era, the use of photons to replace electrons for computing, especially for specialized purposes, has once again come into focus. Theoretically, light signals carrying data travel at the speed of light, a characteristic that makes it possible to design an ultra-low latency computing platform using optical structures. Secondly, the modulation methods of light waves, such as wavelength, phase, polarization, and amplitude, provide rich multiplexing degrees of freedom for optical computing. Furthermore, compared to electrons, photons do not have ohmic losses, resulting in lower energy consumption, and the higher bandwidth they can provide is also highly beneficial for accelerating neural networks. Therefore, due to their high speed, low latency, high parallelism, low energy consumption, and high bandwidth, photons have enormous application potential, at least in the field of specialized computing, especially in massive matrix linear computations for artificial neural networks.
[0003] In artificial neural networks, tensor (multi-channel matrix) computation accounts for over 80% of the total computational power. Tensors, defined in geometric algebra, are generalizations of vectors and matrices. Simply put, scalars can be considered zero-order tensors, vectors first-order tensors, two-dimensional matrices second-order tensors, and multi-channel two-dimensional matrices third-order tensors. Tensor computation in artificial neural networks primarily refers to the convolution operation between the feature matrices of multiple input channels and multiple convolution kernels. In traditional computers, one of the most common implementations of convolution is the Im2Col transform (image to column), which transforms the complex tensor convolution operation into the multiplication of two large matrices. This method consumes computational and storage resources for matrix zeroing and rearrangement; furthermore, because the convolution sliding steps often overlap, the final rearranged matrix is larger than the original input matrix, requiring additional storage resources and ultimately hindering effective energy consumption reduction. The larger the convolution kernel, the more overlapping elements exist between convolution sliding steps, leading to a rapid increase in additional storage resource consumption. In contrast, optical matrix computation can utilize the optical cutoff effect to achieve natural zero-padding. Furthermore, leveraging the inherent parallelism of optics, it can directly achieve time-synchronized multi-step translation, sliding, multiplication, and addition operations in convolution computation. Therefore, the input matrix and the original matrix are of equal size, requiring no additional resource consumption, thus possessing a natural advantage for tensor convolution computation. In this patent, we propose a time-synchronized multi-channel matrix parallel convolution (i.e., tensor convolution) computation accelerator, providing a new technical solution for special-purpose matrix-specific optical computation in the era of artificial intelligence. Summary of the Invention
[0004] To address the issue of excessive resource consumption in tensor convolution computation in electronic computing, this invention proposes an optical tensor computation accelerator and its computation method based on multi-imaging projection. By extracting and rearranging the matrix elements and convolution kernel elements of the input N-channel (N is a positive integer) feature map, both the input feature map and the convolution kernel are transformed into a two-dimensional matrix. Wavelength division multiplexing and polarization multiplexing are then used to achieve the output of the multi-channel convolution kernel, thus realizing tensor computation in a fully optical sense. The optical tensor computation implemented in this invention fully utilizes the parallel characteristics of optics, exhibiting high efficiency, low latency, low power consumption, and no need for additional storage resources, making it of significant practical value in the future development of optical neural networks.
[0005] The technical solution of this invention:
[0006] An optical tensor computation accelerator based on multi-imaging projection is characterized by comprising: a multi-channel optical signal input module, an imaging projection module, a multi-channel optical signal detection module, and the optical tensor computation acceleration module.
[0007] The multi-channel optical signal input module consists of a light source controller, a light source array, and a polarization beam splitter (PBS). Through wavelength division multiplexing and polarization multiplexing, it simultaneously loads multi-channel convolution kernel matrix information onto optical signals with different characteristics.
[0008] The imaging projection module includes multiple devices with beam splitting and diffraction functions and imaging functions. The beam splitting and diffraction devices split the optical signal carrying multi-channel convolution kernel information into multiple sub-beams with different angles and directions. The imaging system then projects multiple displacement images onto its conjugate plane with a magnification factor of ×1. The optical modulator is located on this conjugate plane, thereby realizing the dot product operation of matrix elements between the feature values and the weight values in the convolution kernel.
[0009] The multi-channel optical signal detection module includes a lens, a polarizing beam splitter (PBS), a dichroic mirror, and a photodetector. The lens is used to converge optical signals of different diffraction orders to different positions at specific diffraction angles for summation. The polarizing beam splitter (PBS), dichroic mirror, and photodetector are used to filter out optical signals with different characteristics after summation, thereby simultaneously outputting the multi-channel optical tensor calculation results.
[0010] The optical tensor computation acceleration module includes a light source array and an optical modulator, used to load re-encoded convolution kernel matrix information and feature map matrix information.
[0011] Preferably, the light source array in the multi-channel optical signal input module is an active-emitting vertical-cavity surface-mount laser array (VCSEL), a digital projection processor (DLP), or a fiber array.
[0012] Preferably, the light modulator in the imaging projection module includes a liquid crystal spatial light modulator (SLM) or a digital micromirror array (DMD).
[0013] Preferably, the polarization beam splitter (PBS), dichroic mirror, and photodetector in the multi-channel optical signal detection module can be replaced by other devices with polarization and wavelength detection capabilities.
[0014] The optical tensor calculation acceleration method includes the following steps:
[0015] Step 1: The input N-channel feature map matrices are as follows: , , , of which The elements in the matrix are ,..., ,..., (j=1, 2...N), with a total of h×w elements. Extract the elements from the first row and first column of the N matrices, i.e. Fill in the N elements sequentially and rearrange them into... A square matrix of order 1, denoted as ( This indicates rounding up. At this point, elements in the matrix may be missing, therefore it is necessary to round (...). × -N) missing positions are padded with zeros. The same extraction and permutation is performed on the remaining elements of the N matrices to obtain... , , This generates a two-dimensional matrix A containing multi-channel feature map information, where the height is h×. Width is w× .
[0016] Step 2: The convolution kernel matrices for the N input channels corresponding to each output channel are as follows: , , , of which The elements in the matrix are ,..., ,..., (j=1, 2...N), with a total of k×k elements. Extract the elements from the first row and first column of the N matrices, i.e. ,..., Rearrange the N elements in the same order as in step 1. A square matrix of order 1, denoted as ( (This indicates rounding up). Similarly, missing elements in the square matrix are padded with zeros. By performing the same extraction and permutation on the remaining elements of the N matrices, we can obtain... , , This generates a two-dimensional matrix B containing information from the multi-channel convolution kernel, where the height is k× Width is k× .
[0017] Step 3: Load the re-extracted and rearranged 2D matrix feature map A and convolution kernel B onto the optical modulator array and light source array, respectively. Adjust the sliding step size of convolution kernel B by adjusting the distance between the beam splitter and the light source array. ×s, where This indicates rounding up, where s is the original convolution stride, enabling optical tensor convolution calculations for multi-channel output.
[0018] Technical effects of the present invention:
[0019] By rearranging the input multi-channel original feature maps and convolution kernel matrix elements, multi-channel matrix convolution operations are rapidly performed within the same time period. Simultaneously, wavelength division multiplexing (WDM) and polarization multiplexing techniques are used to output the multi-channel convolution results, demonstrating excellent parallel processing capabilities and making it well-suited for optical neural network architectures. Furthermore, the entire tensor computation process is performed at almost the speed of light, resulting in extremely high computational speeds, far exceeding those of traditional electronic computers. In addition, this method offers simple manipulation of input matrix elements, strong scalability, and greatly meets the computational needs of massive amounts of data in future artificial intelligence, demonstrating significant practical value. Attached Figure Description
[0020] Figure 1 This is a schematic diagram of the multi-channel matrix generating two-dimensional matrix of the present invention.
[0021] Figure 2 This is a schematic diagram of the quad-core, four-channel matrix (tensor) convolution calculation of the present invention.
[0022] Figure 3 This is a schematic diagram of the optical tensor convolution computation accelerator based on multi-imaging projection of the present invention.
[0023] Figure 4 This is a schematic diagram of the light source array in the multi-channel optical signal input module of the present invention.
[0024] Figure 5 This is a schematic diagram of the photodetector in the multi-channel optical signal detection module of the present invention. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and preferred embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0026] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the multi-channel matrix generating two-dimensional matrix of the present invention. By extracting and rearranging the elements of the input multi-channel matrix in positional order, a two-dimensional matrix containing all feature maps or convolutional kernel information is generated.
[0027] Please see Figure 2 ,like Figure 2 As shown, a four-channel 3×3 feature map matrix is input. Since the number of channels in the convolution kernel must be equal to the number of channels in the input, the dimension of the convolution kernel is 2×2×4×4, with four output channels.
[0028] First, the elements of the four-channel feature map matrix are extracted and rearranged. The specific steps are as follows:
[0029] Step 1: Take the first row and first column of the four channel feature map matrices, i.e. , , , Arrange these four elements into a 2×2 matrix. Similarly, take the first row and second column of the four channel feature map matrices, i.e. , , , Arrange these four elements into a 2×2 matrix. Continue in this manner, traversing each row and column of the feature map elements in order, generating a total of 9 2×2 matrices. Arrange these 9 matrices in order to generate a 6×6 square matrix A. Thus, we have achieved the transformation from a four-channel feature map matrix to a two-dimensional matrix.
[0030] Step 2: Perform the same operation on the four channel matrix elements of convolution kernel 1, taking the first row and first column, i.e., 1, 1, 0, 1, and arranging these four elements into a 2×2 small matrix. Repeat this process to generate four 2×2 small matrices. Arrange these four small matrices in order to generate a 4×4 square matrix B containing the information of the four-channel convolution kernel. The same process applies to convolution kernels 2, 3, and 4.
[0031] Please see Figure 3 , Figure 3 This is a schematic diagram of a multi-channel synchronous output optical tensor convolution calculation accelerator. This embodiment uses a four-core, four-channel matrix. The multi-channel optical signal input module includes a light source controller 101, light source arrays 201, 202, 203, and 204, and polarization beam splitters (PBS) 301 and 302. Through wavelength division multiplexing and polarization multiplexing, it simultaneously loads multi-channel convolution kernel matrix information onto optical signals with different characteristics. The imaging projection module includes multiple devices 401 and 402 with beam splitting and diffraction functions and a device 601 with imaging function, realizing the matrix element-wise multiplication function of eigenvalues and weight values. The multi-channel optical signal detection module includes a lens 603, a polarization beam splitter (PBS) 303, dichroic mirrors 801 and 802, and photodetectors 901, 902, 903, and 904. It is used to converge and filter optical signals with different characteristics to obtain the optical tensor calculation results for multiple output channels. The optical tensor calculation acceleration module includes light source arrays 201, 202, 203, and 204 and an optical modulator 701, which are used to load re-encoded matrix information to accelerate optical tensor calculation.
[0032] First, the light source controller 101 controls four light source arrays 201, 202, 203, and 204, utilizing two sets of light with different wavelengths. , Four different carrier signals are generated by passing through two polarization beamsplitters 301 and 302 respectively. The two-dimensional feature matrix A generated after being extracted and rearranged is loaded onto the transmissive spatial light modulator 701, and the convolution kernel matrices B1, B2, B3, and B4 of the four output channels are loaded onto the light source arrays 201, 202, 203, and 204 respectively. At this time, the modulated light signals carrying different convolution kernel information are split into multiple sub-beams with different angular directions by the devices 401 and 402 with beam splitting and diffraction functions. Next, these sub-beams pass through the beamsplitter 501 and the imaging device 601, which projects multiple displacement images of the convolution kernel matrices B1, B2, B3, and B4 onto their conjugate plane with a magnification factor of ×1. The feature matrix A is located on this conjugate plane.
[0033] Secondly, by adjusting the appropriate distance d between the light source array 201 and the device 401 with beam splitting and diffraction capabilities, the displacement distance of the sub-beams is made equal to the spacing between two units of matrix A (the same applies to other output channels). At this point, each sub-beam acts as a sliding window, and the sliding process of the convolution kernel is automatically executed in a large-scale parallel manner. In this way, once the sub-beam of matrix B1 carrying multi-channel convolution kernel information passes through matrix A, element-wise multiplication can be achieved simultaneously.
[0034] Finally, each sub-beam carrying the element-wise multiplication information of the feature matrix A and the convolution kernel matrix B1 is focused by lens 603 to perform an accumulation operation. A polarizing beam splitter 303 and dichroic mirrors 801 and 802 are placed behind lens 603, and four photodetectors 901, 902, 903, and 904 are placed on the back focal plane. This allows for the separate detection of four different sets of modulated light signals, resulting in the optical tensor convolution calculation results with four channels output synchronously. The specific convolution results are shown below.
[0035] After performing the operation in step 1 on the input four-channel 3×3 feature map matrix, the generated two-dimensional feature matrix A is:
[0036]
[0037] After performing the operation in step 2 on the four input four-channel 2×2 convolution kernel matrices, the generated two-dimensional convolution kernel matrices are as follows:
[0038]
[0039] In matrix convolution calculations, to ensure the output feature map has the same matrix size as the input, we need to add an aperture at the input of the feature map that matches the matrix size. In traditional electronic computers, we generally use padding operations to ensure consistent output size. However, in optical computers, since matrix element multiplication utilizes light intensity information, adding an aperture can achieve the function of filling zero pixels.
[0040] In the first window, the convolution result is:
[0041]
[0042] Adjusting the stride of the convolution kernel to 2 cell spacing, the convolution result in the second sliding window is:
[0043] Following the above steps, the remaining sliding window convolution results are as follows:
[0044]
[0045] Similarly, by synchronously loading the matrix information of convolution kernel 2, convolution kernel 3, and convolution kernel 4 onto the light source array and utilizing wavelength division multiplexing and polarization multiplexing, the resulting multi-channel output convolution results are as follows:
[0046]
[0047]
[0048]
[0049] Please see Figure 4 , Figure 4 This is a schematic diagram of the light source array in the multi-channel optical signal input module. It can be (a) a vertical cavity surface-mount semiconductor laser array (VCSEL), or (b) a fiber array, or other light-emitting element arrays, used to load convolution kernel matrix information onto the input optical signal.
[0050] Please see Figure 5 , Figure 5 This is a schematic diagram of the photodetector in the multi-channel optical signal detection module. After the optical signals of different channels are filtered out by the polarization beam splitter (PBS) and dichroic mirror, a photodetector array (PDA), CCD or other optical signal receiver array can be used at the back focal plane of the lens to collect the optical tensor convolution calculation results.
[0051] The embodiments of the optical tensor computing accelerator based on multi-imaging projection described above are merely one specific implementation of the present invention and should not be construed as limiting the scope of protection of the present invention. It should be noted that those skilled in the art can make various modifications and improvements to the specific implementation details and representative devices proposed in this patent without departing from the basic idea of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.
[0052] In summary, this invention proposes an optical tensor computation accelerator based on multi-imaging projection, which can be widely applied in fields such as accelerating optical tensor computation in artificial neural networks.
Claims
1. An optical tensor computation accelerator based on multi-imaging projection, characterized in that, include: The system includes a multi-channel optical signal input module, an imaging projection module, a multi-channel optical signal detection module, and the optical tensor calculation acceleration module. The multi-channel optical signal input module is used to simultaneously load multi-channel convolution kernel matrix information onto optical signals with different characteristics; The imaging projection module is used to split and diffract the optical signal carrying multi-channel convolution kernel matrix information into sub-beams at different angles, and project them onto the back-end optical modulator through the imaging system to realize the matrix element multiplication function of feature values and weight values. The multi-channel optical signal detection module is used to converge and filter optical signals with different characteristics to obtain optical tensor calculation results for multiple output channels. The optical tensor computation acceleration module is used to extract and rearrange the elements of the input multi-channel feature map matrix and the convolution kernel matrix to generate a two-dimensional matrix A containing multi-channel feature map information, where the height is... , width is And a two-dimensional matrix B containing multi-channel convolution kernel information, where the height is... , width is ; Matrix A is loaded into the optical modulator, and matrix B is loaded into the light source array of the multi-channel optical signal input module. By adjusting the distance between the beam splitter and the light source array, the sliding step size of the convolution kernel is adjusted to... ,in This indicates rounding up, where s is the original convolution stride, thus accelerating multi-channel optical tensor computation simultaneously.
2. The optical tensor computation accelerator based on multi-imaging projection according to claim 1, characterized in that, The multi-channel optical signal input module consists of a light source controller, a light source array, and a polarization beam splitter. Through wavelength division multiplexing and polarization multiplexing, it simultaneously loads multi-channel convolution kernel matrix information onto optical signals with different characteristics.
3. The optical tensor computation accelerator based on multi-imaging projection according to claim 1, characterized in that, The imaging projection module includes multiple devices with beam splitting and diffraction functions and imaging functions. The beam splitting and diffraction devices split the optical signal carrying multi-channel convolution kernel information into multiple sub-beams with different angles and directions. The imaging system then projects multiple displacement images onto its conjugate plane with a magnification factor of ×1. The optical modulator is located on this conjugate plane, thereby realizing the dot product operation of matrix elements between the feature values and the weight values in the convolution kernel.
4. The optical tensor computation accelerator based on multi-imaging projection according to claim 1, characterized in that, The multi-channel optical signal detection module includes a lens, a polarization beam splitter, a dichroic mirror, and a photodetector. The lens is used to converge optical signals of different diffraction orders to different positions according to the diffraction angle for summation. The polarization beam splitter, dichroic mirror, and photodetector are used to filter out optical signals with different characteristics after summation, thereby simultaneously outputting the multi-channel optical tensor calculation results.
5. The optical tensor computation accelerator based on multi-imaging projection according to claim 1, characterized in that, The optical tensor computation acceleration module includes a light source array and an optical modulator, used to load re-encoded convolution kernel matrix information and feature map matrix information.
6. The optical tensor computation accelerator based on multi-imaging projection according to claim 1, characterized in that, The accelerator also includes any one or more of the following: The light source array in the multi-channel optical signal input module is an active-emitting vertical-cavity surface-mount semiconductor laser array, a digital projection processor, or a fiber array. The light modulator in the imaging projection module includes a liquid crystal spatial light modulator or a digital micromirror array. The polarization beam splitter, dichroic mirror, and photodetector in the multi-channel optical signal detection module can also be replaced by other devices with polarization and wavelength detection capabilities.
7. A method for accelerating optical tensor calculations using an optical tensor calculation accelerator based on multiple imaging projections as described in any one of claims 1-6, characterized in that, Includes the following steps: Step 1: The input N-channel feature map matrices are as follows: , of which The elements in the matrix are There are a total of h×w elements. Extract the elements from the first row and first column of the N matrices, i.e. Fill in N elements sequentially and rearrange them into A square matrix of order 1, denoted as This indicates rounding up; when elements in the matrix are missing, it is necessary to... Fill missing positions with 0; perform the same extraction and permutation on the remaining elements of the N matrices to obtain... This generates a two-dimensional matrix A containing multi-channel feature map information, where the height is... , width is ; Step 2: The convolution kernel matrices for the N input channels corresponding to each output channel are as follows: , of which The elements in the matrix are There are k×k elements in total. Extract the elements from the first row and first column of N matrices, i.e. Rearrange the N elements in the same order as in step 1. A square matrix of order 1, denoted as Indicates rounding up; Similarly, by padding missing elements in the square matrix with zeros, and performing the same extraction and permutation on the remaining elements of the N matrices, we can obtain... This generates a two-dimensional matrix B containing multi-channel convolution kernel information, where the height is... , width is ; Step 3: Load the re-extracted and rearranged 2D matrix feature map A and convolution kernel B onto the optical modulator array and light source array, respectively. Adjust the sliding step size of convolution kernel B by adjusting the distance between the beam splitter and the light source array. ,in This indicates rounding up, where s is the original convolution stride, enabling optical tensor convolution calculations for multi-channel output.