Optical matrix-matrix multiplication computation system and method based on multi-imaging projection architecture
The optical matrix-matrix multiplication computation system based on a multi-imaging projection architecture solves the problem of large-scale matrix-matrix multiplication computation, and realizes highly parallel, high-speed, and low-power matrix multiplication operations, which are suitable for deep learning and big data fields.
Patent Information
- Application Number
- CN202210795458.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-06
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-07-06
AI Technical Summary
Existing technologies lack effective optical architectures to achieve large-scale matrix-matrix multiplication calculations. Planar integrated optical waveguide schemes can only perform vector-matrix multiplications, and their computing power has limitations. In practice, spatial diffraction optical schemes are difficult to implement three-dimensional subwavelength optical elements with precise control of complex electromagnetic fields.
An optical matrix-matrix multiplication computation system based on a multi-imaging projection architecture is adopted, including a light source array module, an imaging projection module, a signal modulation module, and a detection module. By generating sub-beams with different displacement step sizes and diffraction angle spectra in two orthogonal directions (x and y), parallel multiplication operations of matrix A and matrix B are realized, and the summation operation is performed in the detection module.
It achieves highly parallel, high-speed, and low-power matrix A and matrix B multiplication operations, and obtains the matrix multiplication results directly at the probe module end face, which improves the calculation accuracy and system computing power. It is suitable for large-scale parallel and high-accuracy analog and digital matrix-matrix multiplication.
Smart Images

Figure CN115237201B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of optical computing technology, and more specifically, to an optical matrix-matrix multiplication computing system and method based on a multi-imaging projection architecture. Background Technology
[0002] With the rapid development of deep learning frameworks, models, and algorithms, a series of hardware architectures and acceleration solutions for accelerating deep learning have been researched to meet the stringent computational requirements of deep learning. It is worth noting that approximately 80% of the computations in deep learning involve matrix convolution calculations between neural network layers. However, most electronic computers and various AI accelerators cannot directly implement matrix convolution operations. Therefore, most deep learning frameworks convert matrix convolution calculations into matrix-matrix multiplication calculations, and then use AI accelerators to accelerate vector-matrix multiplication calculations, thereby accelerating deep learning tasks. Because these AI accelerators first convert matrix convolution calculations into matrix-matrix multiplication calculations, and then successively implement the acceleration effect according to vector-matrix multiplication, this greatly increases the amount of data accessed to memory and the waiting time. This is a bottleneck in the architecture of electronic AI accelerators. However, optical computing, due to its inherent parallelism, ultra-fast propagation speed, extremely low power consumption, and absence of electromagnetic interference, has experienced a resurgence in research since 2010 and has become a highly promising solution for matrix computing in the post-Moore's Law era.
[0003] Currently, there are two main optical computing schemes. One is a vector-matrix multiplier based on a planar optical waveguide architecture, mainly composed of cascaded Mach-Zehnder interferometers (Prior Art 1: Nat. Photon. 11, 441 (2017)) or microring resonators (Prior Art 2: Nature 569, 208 (2019), Prior Art 3: Nature 589, 52 (2021), Prior Art 4: Nature 589, 44 (2021)). This technology modulates the signal to complete the vector-matrix multiplication function by adjusting the waveguide refractive index to change the phase or intensity of the optical signal. The other is a spatial diffractive optical neural network based on diffractive optical elements (Prior Art 5: Science 361, 1004-1008 (2018), Prior Art 6: Nature Photonics). (15, 367-373 (2021)) This system uses 3D printing technology to cascade trained diffractive optical layers to achieve the function of a matrix-matrix multiplication neural network. However, planar waveguide architectures can only implement optical vector-matrix multipliers, and their integration has a significant ceiling, resulting in limited computing power, thus limiting their use to coprocessors in electronic AI accelerators. While diffractive optical computation schemes fully utilize the enormous two-dimensional parallelism of spatial optics, the additional computational load caused by the complex relationships of diffraction somewhat offsets the system's acceleration advantage. Furthermore, this scheme uses cascaded diffractive optical devices to achieve spatial interconnection between different matrix-matrix multiplication layers, theoretically avoiding the obstacle of achieving high-density interconnection in vectors, and fully utilizing the spatial interconnection capabilities of optical systems. Essentially, this multi-layered cascaded diffractive optical element is a three-dimensional subwavelength optical element capable of precisely modulating complex electromagnetic fields. However, the practical implementation of such a subwavelength three-dimensional optical device faces significant challenges.
[0004] In fact, space optical computing systems, due to their inherent two-dimensional parallel characteristics, were extensively studied in the 1980s and 1990s, resulting in various signal processing systems with practical value. In 1992, Zhou Changhe et al. implemented a binary optical vector-matrix multiplier based on the shadow projection method (Prior Art 7: Opt. Lett. 17, 1800-1802 (1992)). However, this scheme is essentially still a matrix convolution computing architecture, only implementing the vector-matrix multiplier by rearranging matrix A. This results in many locations on the probe plane containing useless information, leading to a waste of computing power. In addition, in the traditional shadow projection method, the information of matrix A is directly projected onto the plane containing matrix B to realize the multiplication of matrices A and B. There is a characteristic distance d0 = ∆ / (tanθ) between the planes containing matrices A and B. p ), where ∆ is the actual physical size of the pixels in planes A and B of the matrix, and θ pρ represents the angle between light of different diffraction orders and the z-axis, where p = ±1, ±2, ... . Therefore, diffraction interference introduced by the propagation of the light field is unavoidable. This means that the smaller the matrix pixels, the more pronounced the diffraction effect and the worse the computational accuracy. Although increasing the size of the matrix pixels can further reduce the error in the computational results, and a larger pixel size can also reduce the alignment error of all pixels on the plane containing matrices A and B, and the signal-to-noise ratio is sufficiently high, the larger the matrix unit, the larger the distance d, and the more significant the effect of the diffraction. Therefore, there is a severe constraint between the computational power and accuracy of the traditional shadow delivery method's optical computing architecture, making it impossible to achieve large-scale matrix-matrix multiplications, which severely limits the computational power and accuracy of this scheme.
[0005] In summary, there is currently no effective architecture or method to realize large-scale optical matrix-matrix multiplication. Planar integrated optical waveguide schemes can only perform vector-matrix multiplication, and there is a clear ceiling to computing power. Although spatial diffraction optics schemes can realize large-scale matrix calculations in principle, in practice, how to realize three-dimensional subwavelength optical elements with precise control of complex electromagnetic fields, especially in the optical band, still faces insurmountable difficulties. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention provides an optical matrix-matrix multiplication calculation system and method based on a multi-imaging projection architecture.
[0007] An optical matrix-matrix multiplication calculation system and method based on a multi-imaging projection architecture, characterized in that it includes: an optical matrix-matrix multiplication component, wherein the optical matrix-matrix multiplication component sequentially comprises: a light source array module, an imaging projection module, a signal modulation module, and a detection module according to the optical path propagation direction; wherein:
[0008] The light source array module is used to load information of matrix A onto the input optical signal to obtain an optical signal carrying information of matrix A.
[0009] The imaging projection module is used to generate sub-beams with different displacement step sizes and different diffraction angle spectra in two mutually orthogonal directions, x and y, and output them to the signal modulation module.
[0010] The signal modulation module is used to load matrix B information and multiply the information of matrix A carried by the sub-beams with different displacement step sizes and different diffraction angle spectra with the information of matrix B, so as to obtain the information after multiplying the row of matrix A with the corresponding column of matrix B at different positions, and then obtain the optical signal carrying the multiplication information.
[0011] The detection module is used to converge the optical signals carrying the multiplication information, and perform a summation operation on the multiplication information to obtain the multiplication information of matrix A and matrix B.
[0012] The optical matrix-matrix multiplication calculation system and method based on a multi-imaging projection architecture is characterized in that the imaging projection module includes multiple beam splitters, which generate several sub-beams in two mutually orthogonal directions (x and y) from the optical signal carrying matrix A information, forming different displacement step sizes and diffraction angle spectra; wherein each sub-beam is incident on different positions of the signal modulation module according to its own diffraction angle, realizing parallel translation and replication operations of the optical signal carrying matrix A information; the matrix A information carried in each sub-beam incident on the signal modulation module is multiplied with the matrix B information at the pixel level, realizing parallel multiplication operations of the matrix A information and the matrix B information.
[0013] Preferably, the parallel multiplication operation between the information of matrix A and the information of matrix B includes:
[0014] The matrix A information is translated and copied in the x direction, and the movement step size is controlled to overlap the different rows or columns of matrix A to the same row or column. Then the overlapping part is truncated to obtain matrix A' containing the overlapping information.
[0015] The matrix A' containing overlapping information is translated and copied in the y direction orthogonal to x, and the movement step size is controlled to make the matrix A' containing overlapping information completely offset in the y direction, thus obtaining A'' containing overlapping information;
[0016] According to the matrix multiplication operation rules, the matrix A'' containing overlapping information is aligned and multiplied with the matrix B, and then the multiplication information of the matrix A'' and matrix B containing overlapping information is obtained at the corresponding positions;
[0017] The multiplication information propagates along diffraction angle spectra in the x and y directions until the multiplication information is completely separated from each other.
[0018] The summation operation on the multiplied information includes:
[0019] The completely separate sub-beams carrying information about the multiplication of the corresponding row or column of matrix A with the column or row of matrix B are converged to different positions on the detection surface of the detection module according to their respective diffraction angle spectra, thereby realizing the summation operation of the multiplication information.
[0020] Preferably, the plane where the light source array module is located and the plane where the signal modulation module is located satisfy the object-image conjugate relationship.
[0021] Preferably, by adjusting the characteristic distance d between the imaging projection module and the light source array module and the beam splitting ratio of the beam splitter, the scattering angles θx and θy of each diffraction order of the light source array module and the imaging projection module are matched, thereby adjusting the displacement step size of the matrix A information relative to the matrix B in the x and y directions.
[0022] Preferably, the light source array module and the signal modulation module are misaligned with different displacement step sizes in the two orthogonal directions x and y, so that the matrix A information partially overlaps the information of different rows or columns in the x direction, and then completely misaligns the overlapped information in the y direction.
[0023] Preferably, the system further includes any one or more of the following:
[0024] Preferably, the light source array module is an array of light-emitting elements or a spatial light modulator;
[0025] Preferably, the imaging projection module includes one or more Damman gratings, wherein the Damman gratings are one-dimensional Damman gratings or two-dimensional Damman gratings;
[0026] Preferably, the imaging projection module employs a metasurface beam splitter or a metamaterial beam splitter;
[0027] Preferably, the signal modulation module employs a spatial light modulator;
[0028] Preferably, the detection module includes a lens and a photodetector array, wherein the lens is used to converge optical signals of different diffraction orders to different positions according to a specific diffraction angle to perform a summation operation, and the photodetector array is used to detect and photoelectrically convert the matrix C information of the matrix-matrix multiplication result.
[0029] The hardware components include: a precision displacement platform, a servo drive motor, a data acquisition card, and a clock synchronization device. The servo drive motor is used to control the movement step size of the precision displacement platform. The precision displacement platform is used to control the multi-dimensional precision adjustment of all functional modules in the optical matrix-matrix multiplication component. The data acquisition card is used to acquire the matrix information of the matrix-matrix multiplication result detected in the detection module. The clock synchronization device is used to realize the synchronous uploading and downloading of loaded information in the light source array module, signal modulation module, and detection module.
[0030] The control components include: a LabVIEW driver module, a C language driver module, and a microcontroller control module. The LabVIEW driver module provides a graphical human-computer interaction interface, the C language driver module defines the operation instructions for the hardware components, and the microcontroller control module controls the hardware components according to the operation instructions defined by the C language driver module.
[0031] According to a second aspect of the present invention, an optical matrix-matrix multiplication calculation method based on a multi-imaging projection architecture is provided, comprising:
[0032] The input optical signal is loaded with matrix A information to obtain an optical signal carrying matrix A information;
[0033] Optical signals with different displacement step sizes and different diffraction angle spectra are generated in the two mutually orthogonal directions x and y for the optical signal carrying matrix A information.
[0034] Load matrix B information, and multiply the optical signal carrying matrix A information with different displacement step size and different diffraction angle spectrum with matrix B information to obtain the multiplication information of corresponding elements of matrix A information and matrix B information at different positions, and then obtain the optical signal carrying the multiplication information.
[0035] The optical signals carrying the multiplication information are converged to perform a summation operation on the multiplication information, resulting in the multiplication information of matrix A and matrix B.
[0036] Preferably, the optical signals of different diffraction orders in the x-direction are at different angles θ. x The optical signals of different diffraction orders in the y-direction are propagated and controlled by a characteristic distance d so that information from different rows or columns of matrix A partially overlaps in the same row or column, with the optical signals at different diffraction orders in the y-direction at different angles θ. y The propagation and control of the characteristic distance d completely separate the matrix A information that overlaps in the same row or column in the y direction. The sub-beams containing the overlapping information are transmitted to different positions of matrix B according to their respective diffraction angle spectra. The optical signals of each diffraction order that are completely separated in the y direction carry all the information of the matrix A that partially overlaps in the same row or column.
[0037] Performing a multiplication operation between the information in matrix A and the information in matrix B includes:
[0038] The matrix A information is translated and copied in the x direction, and the movement step size is controlled to overlap the different rows or columns of matrix A to the same row or column. Then the overlapping part is truncated to obtain matrix A' containing the overlapping information.
[0039] The matrix A' containing overlapping information is translated and copied in the y direction orthogonal to x, and the movement step size is controlled to make the matrix A' containing overlapping information completely offset in the y direction, thus obtaining the matrix A'' containing overlapping information;
[0040] According to the matrix multiplication operation rules, the matrix A'' containing overlapping information is aligned and multiplied with the matrix B to obtain the overlapping multiplication information of matrix A'' and matrix B;
[0041] The overlapping multiplication information propagates along different diffraction angle spectra in the x and y directions until the overlapping multiplication information is completely separated from each other.
[0042] The summation operation on the multiplied information includes:
[0043] The completely separate sub-beams carrying information about the multiplication of the corresponding row or column of matrix A with the column or row of matrix B are converged to different positions on the detection surface of the detection module according to their respective diffraction angle spectra, thereby realizing the summation operation of the multiplication information.
[0044] Preferably, multiplying the elements in matrix A with the corresponding elements in matrix B includes:
[0045] Perform a multiplication operation between the elements in matrix A and the corresponding elements in matrix B:
[0046] ;
[0047] A mn •B nm =C nn ;
[0048] Among them, [a 11 a 12 , ..., a nm ] represents an element of matrix A, [b 11 b 12 , ..., b nm [ ] represents an element of matrix B, where the subscripts n and m are the row and column numbers of matrices A, B, and C, respectively.
[0049] The elements in matrix A of the multiplication information are multiplied by the corresponding elements in matrix B. The summation operation is then performed.
[0050] c 11 = [a 11 , a 21 , …, a n1 ] × [b 11 , b 12 , …, b n1 ], c12 = [a 11 , a 21 , …, a n1 ]× [b 12 , b 22 , …, b n2 ], …;
[0051] c 21 = [a 21 , a 22 , …, a n2 ] × [b 12 , b 22 , …, b n2 ], c 22 = [a 21 , a 22 , …, a n2 ]× [b 12 , b 22 , …, b n2 ], …;
[0052] …,
[0053] c n1 = [a 1m , a 2m , …, a nm ] × [b 11 , b 12 , …, b n1 ], c n2 = [a 1m , a 2m , …, a nm ] ×[b 12 , b 22 , …, b n2 ], …;
[0054] Preferably, the loaded matrix A information, the loaded matrix B information, and the obtained matrix-matrix multiplication result matrix C information are all analog quantities. The matrix-matrix multiplication result matrix C information is quantized into a digital result after digital processing to realize analog optical matrix-matrix operations.
[0055] Preferably, the high-bit matrix A and high-bit matrix B to be processed are represented as multiple encoded low-bit matrices to obtain encoded low-bit matrix A and low-bit matrix B; the low-bit matrix A and low-bit matrix B are used as loaded matrix information to obtain low-bit matrix-matrix multiplication result matrix information; the low-bit matrix-matrix multiplication result matrix information is decoded into high-bit matrix-matrix multiplication result matrix C information to realize digital optical matrix-matrix operations.
[0056] The optical matrix-matrix multiplication calculation system based on the multi-imaging projection architecture can be optimized and developed based on any form of optical system that satisfies the object-image conjugate relationship between matrix A and matrix B;
[0057] The optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture can further improve computing power through novel optical communication technology that expands capacity. Its key feature is that the light source array module simultaneously loads information from two or more matrices A using two or more optical signals with different characteristics. These optical signals' characteristics include, but are not limited to, wavelength, mode, and polarization. After passing through the imaging projection module and signal modulation module, the various optical signals carrying multiple matrix-matrix multiplication information are multiplied by matrix B. Finally, the detection module detects the results of multiple multiplications of matrices A and B, thus achieving a significant increase in the computing power of optical matrix calculations.
[0058] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:
[0059] The optical matrix-matrix multiplication computation system and method based on a multi-imaging projection architecture provided by this invention can perform the multiplication of matrix A and matrix B in a highly parallel, high-speed, and low-power manner, and directly obtain the matrix multiplication result matrix C on the end face of the detection module (detector). Based on the system and method provided by this invention, analog and digital matrix-matrix multiplications with large-scale parallelism and sufficiently high accuracy can be efficiently calculated. Moreover, the multiplication is universal, and the obtained calculation results are very easy to port to any other computing platform. By developing a high-speed spatial light modulator with higher contrast, optimizing a dedicated projection imaging system, and configuring a dedicated dot matrix light source, an optical matrix-matrix computation system with higher computing power and lower energy consumption compared to an electronic computer can be constructed. In addition, due to the characteristics of the imaging system itself, by cascading multiple 4F systems and using additional multiplexing degrees of freedom, the computing power of the system can be increased exponentially, which is expected to enable the construction of a hybrid optoelectronic computer center or data center based on an optical matrix-matrix multiplier with a multi-imaging projection architecture.
[0060] The optical matrix-matrix multiplication computation system and method based on a multi-imaging projection architecture provided by this invention significantly improves the pixel utilization rate of the spatial light modulator in the optical matrix-matrix multiplication computation system, reduces the requirements for the dynamic detection range of the detection module, and can further improve the computational accuracy of the optical matrix-matrix multiplication system. Compared with electronic AI accelerators and other optical computing solutions, the system and method provided by this invention can realize general-purpose, arbitrary-base digital optical matrix-matrix multiplication computation, and has the characteristics of high speed, low power consumption, high parallelism, high tolerance, large scale, and reconfigurability. The system and method provided by this invention lay the research foundation for digital optical computing and are expected to further develop digital optical computing systems based on matrix transformation, decomposition, and other operations, which have significant application value and good economic benefits in deep learning and other fields involving a large number of matrix operations.
[0061] The optical matrix-matrix multiplication calculation system and method based on a multi-imaging projection architecture provided by this invention is an optical matrix-matrix multiplication calculation system with truly large-scale parallelism and sufficiently high precision. After loading matrices A and B into the input module, the calculation result matrix C of large-scale, high-precision optical matrix multiplication can be directly obtained in the detection module after the optical signal passes through the system once. The planes containing matrices A and B satisfy the object-image conjugate relationship, avoiding the diffraction problem present in traditional shadow projection methods. By introducing an imaging projection module, each diffraction order sub-beam carrying all the information of matrix A is perfectly imaged onto the corresponding position in the plane of matrix A, realizing parallel multiplication operations. Parallel summation operations are achieved through lenses. Furthermore, by adjusting the distance d between matrix A and the imaging projection module and the diffraction angle θ of the beam splitter... x and θ y A perfect match is achieved to modify the parameters for matrix-matrix multiplication calculations. The system and method provided by this invention solve the problems of various applications requiring the calculation of massive amounts of large-scale matrix-matrix multiplications, and have significant economic importance and development prospects in emerging industries involving the national economy and comprehensive national strength, such as big data, artificial intelligence, the Internet of Things, and Industry 4.0. Attached Figure Description
[0062] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0063] Figure 1This is a schematic diagram illustrating the principle of multiplication operation implemented in this invention; where 101 is matrix A, 102 is matrix B, 103 is matrix C, 101a is the first row of matrix A, 101b is the second row of matrix A, 101m is the m-th row of matrix A, 102a is the first column of matrix B, 102b is the second column of matrix B, 102m is the m-th column of matrix B, 103a is the first row of matrix C, 103b is the second row of matrix C, and 103m is the m-th row of matrix C.
[0064] Figure 2 This is a schematic diagram of the optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture according to the present invention; wherein, 201 is a light source array, 202 is a beam splitter, 203 is a Fourier transform lens L1, 204 is a Fourier transform lens L2, 205 is a beam splitter, 206 is a spatial light modulator, 207 is a cutoff aperture, 208 is a focusing lens L3, and 209 is a detector array.
[0065] Figure 3 This is a schematic diagram of four embodiments of the light source array of the optical matrix-matrix multiplication calculation system based on the multi-imaging projection architecture of the present invention; wherein, 201A is an optical fiber array, 201A0 is a modulator, 201A1 is an optical fiber, 201A2 is an integrated light-emitting end, 201B is an on-chip integrated laser array, 201C is a VCSEL array, and 201D is an LED array.
[0066] Figure 4 This is a schematic diagram of three embodiments of the x-direction beam splitting element of the optical matrix-matrix multiplication calculation system based on the multi-imaging projection architecture of the present invention; wherein, 203A is a metamaterial or metasurface, 203B is a diffractive optical element, and 203C is a micromirror array.
[0067] Figure 5 This is a schematic diagram of three implementation cases of the optical matrix-matrix multiplication calculation system based on the multi-imaging projection architecture proposed in this invention, in the y direction orthogonal to the x direction; wherein, 205A is a metamaterial or metasurface, 205B is a diffractive optical element, and 205C is a micromirror array.
[0068] Figure 6 These are schematic diagrams of three embodiments of the signal modulation module of the optical matrix-matrix multiplication calculation system based on the multi-imaging projection architecture of the present invention; wherein, 209A is an optical fiber array, 209B is a camera, and 209C is a detector array.
[0069] Figure 7 These are schematic diagrams of three embodiments of the optical receiver of the optical matrix-matrix multiplication computation system based on the multi-imaging projection architecture of the present invention; wherein, 210A is an optical fiber array, 210B is a camera, and 210C is a detector array.
[0070] Figure 8 This is a schematic diagram of a preferred embodiment of the optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture of the present invention; wherein, 3001 is a VCSEL array, 3002 is a two-dimensional Damman grating, 3003 is a Fourier transform lens L1, 3004 is a Fourier transform lens L2, 3005 is the image plane of the VCSEL, 3006 is a spatial light modulator, 3007 is a cutoff stop, 3008 is a focusing lens L3, 3009 is a detector array, 3010 is a control system, 3101 is matrix A, 3102 is a vector A rearranged into a column, 3103 is a matrix A consisting of three sub-beams carrying information of the rearranged vector A, with three rows of information partially overlapping in the x-direction and completely separated in the y-direction, 3104 is all the overlapping matrices A after being truncated by the cutoff stop, 3105 is matrix B, and 3106 is matrix C.
[0071] Figure 9 This is a flowchart of the optical matrix-matrix multiplication calculation method based on a multi-imaging projection architecture according to the present invention. In this flowchart, 401 is to design system parameters according to the size of matrices A and B; 402 is to select appropriate devices to build the system according to the system principle and parameters; 403 is to rearrange matrices A and B according to the matrix multiplication principle to maximize the pixel utilization of the light source module, signal modulation module and detection module; 404 is to load matrices A and B onto the light source module and signal modulation module respectively, and collect the matrix multiplication result C on the computer for subsequent processing. Detailed Implementation
[0072] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.
[0073] An embodiment of the present invention provides an optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture, comprising: an optical matrix-matrix multiplication component, wherein the optical matrix-matrix multiplication component includes: a light source array module, an imaging projection module disposed at the rear end of the light source array module, a signal modulation module disposed at the rear end of the imaging projection module, and a detection module disposed at the rear end of the signal modulation module; wherein:
[0074] The light source array module is used to load information of matrix A onto the input optical signal to obtain an optical signal carrying information of matrix A.
[0075] The imaging projection module is used to generate sub-beams with different displacement step sizes and different diffraction angle spectra in two mutually orthogonal directions, x and y, and output them to the signal modulation module.
[0076] The signal modulation module is used to load matrix B information and multiply the information of matrix A carried by the sub-beams with different displacement step sizes and different diffraction angle spectra with the information of matrix B, so as to obtain the information after multiplying the row of matrix A with the corresponding column of matrix B at different positions, and then obtain the optical signal carrying the multiplication information.
[0077] The detection module is used to converge the optical signal carrying the multiplication information, realize the summation operation of the multiplication information, and obtain the matrix-matrix multiplication result matrix C information.
[0078] In a preferred embodiment, the imaging projection module includes multiple beam splitters. These beam splitters generate several sub-beams in two mutually orthogonal directions (x and y) from the optical signal carrying matrix A information, forming different displacement step sizes and diffraction angle spectra. Each sub-beam is incident on different positions of the signal modulation module according to its respective diffraction angle, realizing parallel translation and replication operations of the optical signal carrying matrix A information. The matrix A information carried in each sub-beam incident on the signal modulation module is multiplied pixel-by-pixel with the matrix B information, realizing parallel multiplication operations of the matrix A information and the matrix B information.
[0079] In a preferred embodiment, the parallel multiplication operation between the matrix A information and the matrix B information includes:
[0080] The matrix A information is translated and copied in the x direction, and the movement step size is controlled to overlap the different rows or columns of matrix A to the same row or column, and then the one-dimensional vector of the overlapping part is extracted.
[0081] The one-dimensional vector containing overlapping information is translated and copied in the y direction orthogonal to x, and the movement step size is controlled to make the one-dimensional vector containing overlapping information completely offset in the y direction, thus obtaining a two-dimensional matrix containing overlapping information.
[0082] The two-dimensional matrix containing overlapping information is aligned with matrix B according to the matrix multiplication operation rules, and then the multiplication information of the two-dimensional matrix containing overlapping information and matrix B is obtained at the corresponding positions.
[0083] The multiplication information of the two-dimensional matrix containing overlapping information and matrix B information propagates according to different diffraction angle spectra in the x and y directions until the overlapping multiplication information is completely separated from each other.
[0084] The summation operation on the multiplied information includes:
[0085] The completely separate sub-beams carrying information about the multiplication of the corresponding row or column of matrix A with the column or row of matrix B are converged to different positions on the detection surface of the detection module according to their respective diffraction angle spectra, thereby realizing the summation operation of the multiplication information.
[0086] In a preferred embodiment, the plane where the light source array module is located and the plane where the signal modulation module is located satisfy the object-image conjugate relationship.
[0087] In a preferred embodiment, by adjusting the characteristic distance d between the imaging projection module and the light source array module and the beam splitting ratio of the beam splitter, the scattering angle θ of each diffraction order between the light source array module and the imaging projection module is adjusted. x and θ y Matching is performed, and then the displacement step size of the matrix A information relative to the matrix B in the x and y directions is adjusted.
[0088] In a preferred embodiment, the light source array module and the signal modulation module are misaligned with different displacement step sizes in the two orthogonal directions x and y, so that the matrix A information partially overlaps the information in the same row or column in the x direction, and then completely misaligns the overlapped information in the y direction.
[0089] As a preferred embodiment, the system further includes any one or more of the following:
[0090] In a preferred embodiment, the light source array module is an array of light-emitting elements or a spatial light modulator;
[0091] In a preferred embodiment, the imaging projection module includes one or more Damman gratings, wherein the Damman gratings are one-dimensional Damman gratings or two-dimensional Damman gratings;
[0092] As a preferred embodiment, the imaging projection module employs a metasurface beam splitter or a metamaterial beam splitter;
[0093] In a preferred embodiment, the signal modulation module employs a spatial light modulator;
[0094] In a preferred embodiment, the detection module includes a lens and an optical receiver array. The lens is used to converge optical signals of different diffraction orders to different positions at specific diffraction angles to perform a summation operation. The optical receiver array is used to detect the matrix information of the matrix-matrix multiplication result.
[0095] A preferred embodiment of the present invention provides an optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture. This system can achieve high-precision, large-scale multiplication calculation of C=A×B, where A is the input matrix A, B is the input matrix B, C is the multiplication result matrix, and × represents the multiplication operation. The multiplication operation refers to multiplying the elements in matrix A with the corresponding elements in matrix B and summing them. Matrix A slides on matrix B with a fixed step size and repeats the multiplication and summation operations. Finally, after traversing all positions, the matrix obtained by multiplying the two matrices is obtained.
[0096] The optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture provided in this preferred embodiment includes two parts: an optical matrix multiplication part and an electronic control part. The optical matrix multiplication part includes a light source array module, an imaging projection module, a signal modulation module, and a detection module; the electronic control part includes various hardware components and control components.
[0097] In a preferred embodiment, the imaging projection module employs a large-scale beam splitter to achieve parallel copying, translation, and multiplication operations of matrix A on matrix B. The optical signal carrying matrix A information is divided into several sub-beams after passing through the large-scale beam splitter. Each sub-beam performs a parallel copying operation with matrix A. Each sub-beam is incident on different positions of the signal modulation module carrying matrix A information according to its respective diffraction angle; this corresponds to a parallel translation operation. The optical signals incident on different positions of the signal modulation module are subjected to pixel-level multiplication of matrix B and matrix A by the signal modulation module; this corresponds to a parallel multiplication operation. The above process can be represented as:
[0098] ;
[0099] A mn •B mn =C nn ;
[0100] Among them, [a 11 a 12 , ..., a nm ] represents an element of matrix A, [b 11 b 12 , ..., b nm [] represents an element of matrix B, where the subscripts n and m are the row or column numbers of matrices A, B, and C, respectively;
[0101] The elements in matrix A of the multiplication information are multiplied by the corresponding elements in matrix B. The summation operation is then performed.
[0102] c 11 = [a 11 , a 21 , …, a n1 ] × [b11 , b 12 , …, b n1 ], c 12 = [a 11 , a 21 , …, a n1 ]× [b 12 , b 22 , …, b n2 ], …
[0103] c 21 = [a 21 , a 22 , …, a n2 ] × [b 12 , b 22 , …, b n2 ], c 22 = [a 21 , a 22 , …, a n2 ]× [b 12 , b 22 , …, b n2 ], …
[0104] …,
[0105] c n1 = [a 1m , a 2m , …, a nm ] × [b 11 , b 12 , …, b n1 ], c n2 = [a 1m , a 2m , …, a nm ] ×[b 12 , b 22 , …, b n2 ], …
[0106] The imaging projection module is positioned after the light source array module and before the signal modulation module. It mainly consists of a large-scale beam splitter and a projection lens group. The large-scale beam splitter is used to achieve parallel translation and replication of the optical signal carrying the information of matrix A on the signal modulation module. The optical signals of different diffraction orders after passing through the large-scale beam splitter propagate at different angles θ. x and θ yThe optical signals of different diffraction orders are transmitted to different positions on the plane of the signal modulation module. Here, x and y are the numbers of the diffraction orders in two mutually orthogonal directions. The optical signal of each diffraction order carries all the information of matrix A and is multiplied with the information of the corresponding position of matrix A loaded on the signal modulation module. The optical signals of different diffraction orders of the large-scale beam splitter are propagated to different positions on the plane of the signal modulation module, which corresponds to the parallel translation and sliding of matrix A on matrix B and the multiplication operation in the multiplication calculation.
[0107] The signal modulation module is placed after the imaging projection module and before the detection module. The signal modulation module is used to modulate the optical signals of different diffraction orders that carry matrix A information after passing through the imaging projection module, so as to realize the multiplication operation between matrix-matrix multiplication kernel B and matrix A. The signal modulation module loads the information of matrix B.
[0108] The detection module is placed after the signal modulation module. The detection module is used to realize the convergence summation and detection of matrix-matrix multiplication results. Specifically, through optical elements such as lenses in the detection module, optical signals carrying the multiplication information of corresponding elements of matrices A and B at different positions are converged to different positions on the detection surface of the detection module according to the tilt components of the diffraction order to realize the summation operation. The lens element in the detection module performs a convergence operation on the optical signals carrying the multiplication information of A and B, which corresponds to the summation operation of the multiplication information of corresponding elements of matrices A and B in the matrix-matrix multiplication calculation.
[0109] The optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture provided in this preferred embodiment can realize both analog and digital optical multiplication operations; wherein:
[0110] When performing analog optical multiplication operations through the optical matrix-matrix multiplication calculation system, matrices A and B are both analog quantities, and the multiplication result matrix C obtained on the detection module is also an analog quantity. After digital processing, the multiplication result matrix C is quantized into a digital result.
[0111] When performing high-precision, large-scale digital optical multiplication operations using an optical matrix-matrix multiplication calculation system, the high-bit matrix to be processed is first represented as multiple encoded low-bit matrices according to matrix encoding and decoding algorithms. Then, the encoded low-bit matrices A and B are loaded onto the light source array module and the signal modulation module, respectively. When light passes through the system once, the low-bit multiplication result matrix C is obtained on the detection surface of the detection module. Finally, the low-bit multiplication result matrix is decoded into a high-bit matrix.
[0112] In a preferred embodiment, the electronic control section includes hardware components and control components. The hardware components include a precision displacement platform, a servo drive motor, a data acquisition card, and a phase-locked loop (PLL) synchronization device. The servo drive motor controls the movement step size of the precision displacement platform, which controls the multi-dimensional precision adjustment of all optical elements in the optical matrix-matrix multiplication section. The data acquisition card is used to acquire the matrix-matrix multiplication result matrix detected by the detection module at high speed. The PLL synchronization device is used to synchronously upload and download the loaded information in the light source array module, signal modulation module, and detection module. The control components include a LabVIEW driver module, a C language driver module, and a microcontroller control module. The LabVIEW driver module is used to complete the graphical human-computer interaction interface, the C language driver module is used to define the operation instructions for the hardware components, and the microcontroller control module completes the control of all hardware components according to the operation instructions defined by the C language driver module.
[0113] As a preferred embodiment, the light source array module can be an array of light-emitting elements, including but not limited to LEDs, LDs, fiber arrays, vertical cavity surface-mount semiconductor laser arrays (VCSELs), or spatial light modulators (SLMs), including but not limited to liquid crystal spatial light modulators (LCSLMs), digital micromirror arrays (DMDs), microelectromechanical systems (MEMS), fiber arrays, and optical waveguide arrays.
[0114] As a preferred embodiment, the large-scale beam splitter can be a Dammann grating with a beam splitting ratio of 1×3 to 1×128, or a two-dimensional Dammann grating with a beam splitting ratio of 3×3 to 128×128 or larger. By combining Dammann gratings, higher diffraction efficiency and larger-scale beam splitting effect can be achieved. Other beam splitting elements can also be used, including but not limited to diffractive optical elements (DOE), low-dimensional functional materials, metasurfaces, etc.
[0115] As a preferred embodiment, the signal modulation module includes, but is not limited to, various spatial light modulators (SLMs), such as liquid crystal spatial light modulators (LDSLMs), digital micromirror arrays (DMDs), microelectromechanical systems (MEMS), fiber arrays, and optical waveguide arrays.
[0116] In a preferred embodiment, the detection module includes a lens and an optical receiver array. The lens is used to converge and sum optical signals of different diffraction orders at different locations, and the optical receiver array is used to detect optical matrix-matrix multiplication signals, including but not limited to CMOS, CCD and various photodetector arrays.
[0117] The optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture provided in this embodiment can be optimized and developed based on any form of optical system that satisfies the object-image conjugate relationship between matrix A and matrix B. It can be a transmissive optical system, a refractive optical system, or a reflective optical system. It can be a paraxial optical system, an off-axis optical system, or a planar space optical waveguide system. It includes, but is not limited to, various variations and combinations of single or multiple cascaded 4F canonical signal systems, microscopic systems, telescope systems, projection systems, and the above optical systems.
[0118] Computing power can be further improved by new optical communication technologies that can expand capacity, including but not limited to wavelength division multiplexing, mode division multiplexing, and polarization multiplexing.
[0119] The detection module is located after the signal modulation module. It is used to achieve the convergence and summation of matrix-matrix multiplication results and to detect the results. Through optical elements such as lenses in the detection module, the optical signals containing the multiplication information of corresponding elements of matrices A and B at different positions are converged to different positions on the detection surface of the detection module according to their diffraction orders to achieve the summation operation. The lens elements in the detection module converge the optical signals carrying the multiplication information of A and B, corresponding to the summation operation of the corresponding element-wise multiplication information of matrices A and B in the matrix multiplication calculation. The detection module includes lenses and an optical receiver array. The lenses are used to converge and sum the optical signals of different diffraction orders at different positions, and the optical receiver array is used to detect the optical matrix multiplication signals, including but not limited to CMOS and CCD detectors.
[0120] Compared to traditional shadow projection methods, the optical matrix-matrix multiplication computation system based on a multi-imaging projection architecture provided in the above embodiments of the present invention effectively utilizes diffraction effects to achieve large-scale matrix-matrix multiplication computations. This is because the planes containing matrices A, B, and the detection module have object-image conjugate relationships with each other. When a diffractive optical element (DOE) such as a Dammann grating is placed in front of the plane of matrix A, the object-image relationships of each plane are still satisfied, and each diffraction order involved in the DOE matrix-matrix multiplication is clearly imaged. Therefore, the physical size of the matrix elements can be greatly reduced. Even commercial optical imaging systems can easily achieve high-quality imaging of matrix elements with sizes on the order of tens of μm, which makes this architecture have great potential to realize a highly integrated miniature optical matrix-matrix multiplication computer with ultra-high computing power and ultra-low power consumption, comparable to a handheld camera. In addition, the higher orders of the Dammann grating can be completely filtered by the aperture without affecting the calculation results. Furthermore, by optimizing and combining Damman gratings with different splitting ratios and high diffraction efficiencies, it is possible to achieve highly efficient spectral splitting of a large-scale matrix with almost no energy loss.
[0121] Compared to the various technical solutions proposed so far, the optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture provided by the above embodiments of the present invention makes full use of optical two-dimensional parallelism and effectively solves the problems of low accuracy and small scale in the traditional shadow delivery method architecture. It has the advantages of good tolerance performance, high accuracy, and large scale. In future big data analysis and advanced artificial intelligence application scenarios, especially in the field of image processing with matrix-matrix multiplication neural network technology as the core, it has significant application value and good economic benefits.
[0122] The technical solutions provided by the above embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0123] The optical matrix-matrix multiplication calculation system based on the multi-imaging projection architecture proposed in this invention realizes digital optical matrix-matrix multiplication operations based on the imaging projection method (OMica). Figure 1 This is a schematic diagram of the multiplication operation principle implemented in this invention, including matrix A101, matrix B102, matrix C103, the first row 101a of matrix A, the second row 101b of matrix A, the m-th row 101m of matrix A, the first column 102a of matrix B, the second column 102b of matrix B, the m-th column 102m of matrix B, the first row 103a of matrix C, the second row 103b of matrix C, and the m-th row 103m of matrix C.
[0124] This invention implements the principle of multiplying two matrices A and B, namely, the formula C = A × B. First, matrix A is split by rows. This is because matrix multiplication requires that each row or column 101a, 101b, 101m of matrix A to the left of the "×" be multiplied element-wise with all columns or rows 102a, 102b, 102m of matrix B to the right of the "×". That is:
[0125] (103c).
[0126] Finally, iterate through all positions to obtain the result of matrix multiplication.
[0127] Figure 2This is a schematic diagram of an optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture proposed in this invention. It includes a light source array 201, a beam splitter 202, Fourier transform lenses L1203 and L2204, a beam splitter 205, a spatial light modulator 206, a cutoff stop 207, a focusing lens L3208, and a detector array 209. The light source array 201 generates an optical signal carrying matrix A information. This signal is imaged onto the beam splitter 205 via a 4F imaging system composed of Fourier transform lenses L1203 and L2204 at its rear focal plane. The beam splitter 202 is placed in front of the Fourier transform lens L1203 at a characteristic distance d from the light source array. The beam splitter has different splitting ratios and diffraction angles in the x and y directions. Therefore, by changing d, the sub-beams carrying all the information of matrix A obtained by the beam splitter can partially overlap in the x-direction and completely separate in another direction orthogonal to the x-direction. Then, the cutoff aperture 207 intercepts the optical signal carrying all overlapping information. The spatial light modulator 208 is placed at the image plane 205 of the light source array, and the optical signal carrying all overlapping information is multiplied at the pixel level with the matrix B information loaded on the spatial light modulator 208. Then, the optical signal carrying the pixel-level dot product information of matrices A and B is focused by the focusing lens L3208 to converge the optical signals of different tilt angle components to different positions on the plane of the detector array 209, realizing the summation operation of the pixel-level dot product information of matrices A and B.
[0128] Figure 3 This diagram illustrates four specific implementation examples of the light source array for the optical matrix-matrix multiplication computation system based on a multi-imaging projection architecture proposed in this invention. The array comprises: fiber optic array 201A, modulator 201A0, fiber optic cable 201A1, integrated light-emitting end 201A2, on-chip integrated laser array 201B, VCSEL array 201C, and LED array 201D. Fiber optic array 201A includes three parts: modulator 201A0, fiber optic cable 201A1, and integrated light-emitting end 201A2. The modulator is mainly used to modulate the light source signal, i.e., to load information. The fiber optic cable is used to transmit the optical signal loaded with information. The integrated light-emitting end integrates and encapsulates all the fibers at a fixed spacing to form a two-dimensional light-emitting array. The on-chip integrated laser array can directly modulate the light source output signal using technologies such as quantum well cascaded lasers, microring resonant optical frequency combs, quantum dot lasers, and DFB lasers, or it can achieve modulation of the optical signal using Mach-Zehnder interferometers, photonic crystals, and microring resonant cavities. VCSEL arrays, or vertically emitting lasers, allow for individual control of the optical signal output from each channel. LED arrays primarily include LED light sources such as OLEDs, μLEDs, and MiniLEDs, with each channel's optical signal also individually controllable.
[0129] Figure 4This diagram illustrates three specific implementation examples of the x-direction beam splitter element in the optical matrix-matrix multiplication computation system based on a multi-imaging projection architecture proposed in this invention. The elements are: metamaterial and metasurface 203A, diffractive optical element 203B, and micromirror array 203C. Metamaterial and metasurface 203A allows for the adjustment of characteristic parameters of the optical signal through microstructure modulation, enabling optical signals with different characteristic parameters to carry the same light source information. Diffractive optical element 203B allows optical signals with different angular spectra to carry the same light source information by superimposing different phase information. Micromirror array 203C uses micromirrors in different directions to propagate optical signals carrying the same light source information with different tilt angle components.
[0130] Figure 5 This diagram illustrates three specific implementation examples of the optical matrix-matrix multiplication computation system based on a multi-imaging projection architecture proposed in this invention, along the y-direction orthogonal to the x-direction. The components are a metamaterial and metasurface 205A, a diffractive optical element 205B, and a micromirror array 205C. The metamaterial and metasurface 205A can be designed with microstructures to modulate the characteristic parameters of the optical signal, allowing optical signals with different characteristic parameters to carry the same light source information. The diffractive optical element 205B, by superimposing different phase information, allows optical signals with different angular spectra to carry the same light source information. The micromirror array 205C uses micromirrors in different directions to propagate optical signals carrying the same light source information with different tilt angle components.
[0131] Figure 6 This diagram illustrates three specific implementation examples of the signal modulation module of the optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture proposed in this invention. The components are: fiber optic array 209A, camera 209B, and detector array 209C. The liquid crystal spatial light modulator has a modulation speed of 60Hz, which is relatively slow and has low contrast. The micromirror array and micromechanical array have higher modulation speeds, reaching kHz. All three modulate the optical signal carrying matrix A information, i.e., loading matrix B information. When light passes through the spatial light modulator, pixel-level multiplication of matrices A and B is achieved.
[0132] Figure 7 This diagram illustrates three specific implementation examples of the optical receiver for the optical matrix-matrix multiplication computation system based on a multi-imaging projection architecture proposed in this invention. The system comprises an optical fiber array 210A, a camera 210B, and a detector array 210C. The optical fiber array 210A includes an integrated receiver 210A2 and an optical fiber 210A1. The integrated receiver receives the optical signal carrying the dot product information of matrix A and matrix B and transmits the acquired signal to the subsequent detector for conversion into an electrical signal. The camera mainly includes CCD and CMOS sensors, capable of acquiring complete information, but with an acquisition speed of less than 100 frames / s. The detector array is relatively large and difficult to integrate.
[0133] The optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture provided in the above embodiments of the present invention is described in the following specific optical implementation method of the matrix-matrix multiplication process:
[0134] Figure 8 This is a schematic diagram of a preferred specific implementation of the optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture proposed in this invention. The system includes: VCSEL array 3001, two-dimensional Damman grating 3002, Fourier transform lens L1 3003, Fourier transform lens L2 3004, image plane of VCSEL 3005, spatial light modulator 3006, cutoff aperture 3007, focusing lens L3 3008, detector array 3009, control system 3010, matrix A 3101, vector A 3102 rearranged into a column, three sub-beams 3103 carrying information of the rearranged vector A, matrix A with three rows of information partially overlapping in the x-direction and completely separated in the y-direction, matrix A 3104 after being truncated by the cutoff aperture, matrix B 3105, and matrix C 3106.
[0135] The VCSEL array 3001 has a wavelength of 808nm, a pixel pitch of 250μm, a pixel size of 10μm, an array size of 128×128, and a total power of 6W. First, matrix A is loaded onto VCSEL 3001. Matrix A3101 has a size of 3×3, and the rearranged matrix is 9×13102.
[0136] The VCSEL array 3001 is imaged onto the image plane 3005 of the VCSEL through Fourier transform lenses L13003 and L23004.
[0137] The two-dimensional Dammann grating 3002 is placed 104 mm in front of the imaging face of the VCSEL array 3001. The number of Dammann gratings in the x-direction is 1×14, and the period is 112 μm. The number of Dammann gratings in the y-direction is 1×14, and the period is 112×8 μm.
[0138] The Fourier transform lenses L13003 and L23004 have a focal length of 400mm. The optical signal emitted from the VCSEL array carries matrix A information. After passing through the beam splitter, the sub-beams are misaligned in the x-direction, and information from different columns is superimposed onto the same column 3103. Simultaneously, in the y-direction, these overlapping optical signals are completely separated, thus resulting in three columns of sub-beam signals 3103 carrying overlapping information.
[0139] The cutoff aperture 3007 has a spacing of 0.75 mm, which is used to intercept three columns of sub-beam signals 3103 carrying overlapping information. The intercepted sub-beam light signals propagate forward to the modulation surface 3006 of the spatial light modulator according to their respective diffraction angles, and are then modulated a second time by the information of matrix B 3104 loaded on the spatial light modulator to realize pixel-level dot product operation between corresponding positions of matrix A and matrix B.
[0140] The optical signal carrying the dot product information of matrices A and B passes through lens L33008 with a focal length of 150mm and propagates to different positions of detector array 3009 to obtain matrix C3106, realizing the summation operation of matrix A and B information.
[0141] The control system 3010 mainly consists of a hardware system and a software control system. The hardware system includes a computing unit, a data acquisition card, and a phase-locked loop synchronization device. The software system includes LabVIEW driver, C language driver, and microcontroller control software. The hardware system can realize data loading, synchronization, acquisition, digital-to-analog / analog-to-digital conversion, and photoelectric conversion. The software system can realize the automated control of the above hardware and the real-time display and processing of experimental results. The electronic control unit 3011 loads vector A3102 and matrix B3104 onto the corresponding VCSEL array 3001 and spatial light modulator 3006, and controls the two-dimensional Damman grating 3002, Fourier transform lens L13003, Fourier transform lens L23004, VCSEL image plane 3005 and characteristic distance d in the imaging projection module to match the different diffraction angle spectra of the two-dimensional Damman grating 3002 in the x and y directions. When the light signal passes through the system once, the matrix A and B matrix-matrix multiplication result matrix C3106 is directly obtained in the detector array 3009. Then, the result is collected into the computer for processing and the correctly decoded matrix-matrix multiplication result is output, thus completing one optical matrix-matrix multiplication calculation.
[0142] Figure 9This is a flowchart of the optical matrix-matrix multiplication calculation method based on a multi-imaging projection architecture proposed in this invention. The system parameters are designed according to the scale of matrices A and B 401. Appropriate devices are selected to build the system according to the system principle and parameters 402. Matrices A and B are rearranged according to the matrix multiplication principle 403. Matrices A and B are loaded into the light source module and signal modulation module respectively 404. The matrix multiplication result C is collected on the computer for further processing and analysis. The flowchart of the optical matrix-matrix multiplication calculation method based on a multi-imaging projection architecture is as follows: (1) Calculate the system parameters according to the scale of matrices A and B, including the characteristic distance d1 of the Damman grating from the imaging plane of matrix A in the x-direction, the characteristic distance d2 of the Damman grating from the imaging plane of matrix A in the y-direction, the spacing of the cutoff gratings, and the period of the Damman gratings; (2) Fabricate the Damman gratings according to the calculated parameters. And select appropriate optoelectronic devices to build the system; (3) According to the attached... Figure 1 The matrix multiplication principle is shown. Rearrange matrices A and B. (4) Load the rearranged matrices A and B onto the light source module and signal modulation module respectively. When the light passes through the system once, the multiplication result matrix C can be obtained on the detector plane. (5) Collect the multiplication result matrix C on the computer and perform subsequent processing according to the encoding and decoding algorithms.
[0143] In some embodiments of the present invention:
[0144] This optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture can achieve high-precision, large-scale matrix-matrix multiplication calculations of C=A×B, where A is matrix A, B is matrix B, C is the matrix-matrix multiplication result matrix, and × represents the matrix-matrix multiplication operation. The matrix-matrix multiplication operation refers to multiplying the elements in matrix A with the corresponding elements in matrix B and summing them. The matrix-matrix multiplication kernel slides on the other matrix with a fixed step size and repeats the multiplication and summation operations. Finally, it traverses all positions to obtain the matrix after matrix-matrix multiplication of the two matrices.
[0145] The light source array module is placed before the imaging projection module, and the light source array module is used to load the information of matrix A.
[0146] The imaging projection module is positioned after the light source array module and before the signal modulation module. The imaging projection module is used to translate and replicate the optical signal carrying the information of matrix A on the signal modulation module. Specifically, the optical signals of different diffraction orders in the x and y directions, after passing through the imaging projection module, are projected at different angles θ. x(p) and θ y(p)The optical signals of different diffraction orders are transmitted to different positions on the plane of the signal modulation module. Here, x(p) and y(p) are the diffraction order numbers. Each diffraction order optical signal carries all the information of matrix A and is multiplied with the information of the corresponding position of matrix B loaded on the signal modulation module. The optical signals of different diffraction orders of the imaging projection module are propagated to different positions on the plane of the signal modulation module, which corresponds to the translation, sliding and multiplication operations of matrix A on matrix B in matrix-matrix multiplication calculation.
[0147] The signal modulation module is positioned after the imaging projection module and before the detection module. It modulates the optical signals carrying matrix A information after passing through the imaging projection module, and loads matrix B information onto the signal modulation module. The detection module is positioned after the signal modulation module and performs convergence summation and detection of matrix-matrix multiplication results. Through optical elements such as lenses in the detection module, the optical signals containing the element-wise multiplication information of matrices B and A at different positions are converged to different positions on the detection surface of the detection module according to the tilt components of the diffraction order to achieve the summation operation. The lens elements in the detection module converge the optical signals carrying the multiplication information of A and B, corresponding to the summation operation of the element-wise multiplication information of matrices A and B in the matrix-matrix multiplication calculation.
[0148] This optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture can perform both analog and digital matrix-matrix multiplication operations. When performing analog matrix-matrix multiplication calculations, matrices A and B are analog quantities, and the matrix-matrix multiplication result matrix C obtained from the detection module is also an analog quantity. After digital processing, the matrix-matrix multiplication result matrix C is quantized into a digital result. When performing high-precision, large-scale digital optical matrix-matrix multiplication calculations, the system uses matrix encoding and decoding algorithms to represent the high-bit matrix to be processed as multiple encoded low-bit matrices. The encoded low-bit matrices A and B are then loaded onto the light source array module and signal modulation module, respectively. When light passes through the system once, the low-bit matrix-matrix multiplication result matrix C is obtained on the detection surface of the detection module. Finally, the low-bit matrix-matrix multiplication result matrix is decoded into a high-bit matrix.
[0149] The electronic control components section includes hardware components and control components. The hardware components include a precision displacement platform, a servo drive motor, a data acquisition card, and a phase-locked loop synchronization device. The control components include a LabVIEW driver module, a C language driver module, and a microcontroller control module. The hardware components can realize data loading, synchronization, acquisition, digital-to-analog / analog-to-digital conversion, and photoelectric conversion. The control components can realize the automated control of the above hardware components and the real-time display and processing of experimental results.
[0150] The light source array module can be an array of light-emitting elements, including but not limited to LEDs, LDs, fiber arrays, and vertical cavity surface-mount semiconductor laser arrays (VCSELs), or it can be a spatial light modulator (SLM), including but not limited to liquid crystal spatial light modulators (LCSLMs), digital micromirror arrays (DMDs), microelectromechanical systems (MEMS), fiber arrays, and optical waveguide arrays.
[0151] The imaging projection module can be a Dammann grating with a beam splitting ratio of 1×3 to 1×128, or a two-dimensional Dammann grating with a scale of 3×3 to 128×128 or larger. By combining Dammann gratings, higher diffraction efficiency and larger-scale beam splitting effect can be achieved. It can also be other beam splitting elements, including but not limited to diffractive optical elements (DOE), metasurfaces and metamaterials.
[0152] Signal modulation modules include, but are not limited to, various spatial light modulators (SLMs), such as liquid crystal spatial light modulators (LDSLMs), digital micromirror arrays (DMDs), microelectromechanical systems (MEMS), fiber optic arrays, and optical waveguide arrays.
[0153] The detection module includes at least one lens and an optical receiver array. The lens is used to converge and sum optical signals of different diffraction orders at different positions, and the optical receiver array is used to detect optical matrix-matrix multiplication signals, including but not limited to CMOS and CCD detectors.
[0154] The optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture provided in the above embodiments of the present invention can be optimized and developed based on any form of optical system that satisfies the object-image conjugate relationship between matrix A and matrix B. It can be a transmissive optical system, a refractive optical system, or a reflective optical system. It can be a paraxial optical system, an off-axis optical system, or a planar space optical waveguide system. It includes, but is not limited to, various variations and combinations of single or multiple cascaded 4F canonical signal systems, microscopic systems, telescope systems, projection systems, and the above systems.
[0155] The optical matrix-matrix multiplication computing system based on a multi-imaging projection architecture provided in the above embodiments of the present invention can further improve computing power through novel optical communication technologies that expand capacity, including but not limited to wavelength division multiplexing, mode division multiplexing, and partial division multiplexing.
[0156] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the system to implement the steps and flow of the method. That is, the embodiments in the system can be understood as preferred examples of the method, and will not be elaborated here.
[0157] The optical matrix-matrix multiplication calculation system and method based on a multi-imaging projection architecture provided in the above embodiments of the present invention achieves high parallelism and high precision matrix-matrix multiplication calculations of matrices A and B. Using the technical solution provided in the above embodiments of the present invention, by simply loading matrices A and B onto the corresponding light source array module and signal modulation module, a large-scale, high-precision matrix-matrix multiplication result matrix C can be directly obtained on the detection module after the optical signal passes through the optical matrix-matrix multiplication calculation system in a single pass. The optical matrix-matrix multiplication calculation system and method based on a multi-imaging projection architecture proposed in the above embodiments of the present invention can realize a new technical route for large-scale, high-precision, and fully parallel optical computing, providing a general and efficient solution to meet the needs of tasks such as artificial intelligence and neural network image processing for massive matrix-matrix multiplication operations.
[0158] The optical matrix-matrix multiplication calculation system and method based on a multi-imaging projection architecture disclosed above represent only one specific embodiment of the present invention and should not be construed as limiting the scope of protection of the present invention. It should be noted that those skilled in the art can make several non-inventive modifications and improvements to the specific implementation details and representative devices proposed in this patent without departing from the basic idea of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.
[0159] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples, without contradiction. Having described the invention in such detail and with reference to its preferred embodiments, it will be apparent that modifications and variations are possible without departing from the scope of the invention as defined in the appended claims.
[0160] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.
Claims
1. An optical matrix-matrix multiplication computation system based on a multi-imaging projection architecture, characterized in that, include: An optical matrix-matrix multiplication component, comprising, in sequence according to the optical path propagation direction: a light source array module, an imaging projection module, a signal modulation module, and a detection module; wherein: The light source array module is used to load information of matrix A onto the input optical signal to obtain an optical signal carrying information of matrix A. The imaging projection module is used to generate sub-beams with different displacement step sizes and different diffraction angle spectra in two mutually orthogonal directions, x and y, respectively, based on the optical signal carrying matrix A information, and output them to the signal modulation module. The signal modulation module is used to load matrix B information and perform parallel multiplication operations on the information of matrix A carried by the sub-beams with different displacement step sizes and different diffraction angle spectra with the information of matrix B, so as to obtain the information after multiplying the row of matrix A with the corresponding column of matrix B at different positions, and then obtain the optical signal carrying the multiplication information. The detection module is used to converge the optical signal carrying the multiplication information to perform a summation operation on the multiplication information and obtain the multiplication information of matrix A and matrix B. The imaging projection module includes multiple beam splitters, which generate several sub-beams in two mutually orthogonal directions (x and y) from the optical signal carrying matrix A information, forming different displacement step sizes and diffraction angle spectra. Each sub-beam is incident on different positions of the signal modulation module according to its own diffraction angle, realizing parallel translation and replication operations of the optical signal carrying matrix A information. The matrix A information carried in each sub-beam incident on the signal modulation module is multiplied with the matrix B information at the pixel level, realizing parallel multiplication operations of the matrix A information and the matrix B information. The parallel multiplication operation between matrix A and matrix B includes: performing a multiplication operation on matrix A and matrix B, and summing the multiplied information; wherein, the multiplication operation on matrix A and matrix B includes: The matrix A information is translated and copied in the x direction, and the movement step size is controlled to overlap the different rows or columns of matrix A to the same row or column. Then the overlapping part is truncated to obtain matrix A' containing the overlapping information. The matrix A' containing overlapping information is translated and copied in the y direction orthogonal to x, and the movement step size is controlled to make the matrix A' containing overlapping information completely offset in the y direction, thus obtaining A'' containing overlapping information; According to the matrix multiplication operation rules, the matrix A'' containing overlapping information is aligned and multiplied with the matrix B, and then the multiplication information of the matrix A'' and matrix B containing overlapping information is obtained at the corresponding positions; The multiplication information propagates along diffraction angle spectra in the x and y directions until the multiplication information is completely separated from each other; The summation operation on multiplied information includes: The completely separate sub-beams carrying information about the multiplication of the corresponding row or column of matrix A with the column or row of matrix B are converged to different positions on the detection surface of the detection module according to their respective diffraction angle spectra, thereby realizing the summation operation of the multiplication information.
2. The optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture according to claim 1, characterized in that, The plane where the light source array module is located and the plane where the signal modulation module is located satisfy the object-image conjugate relationship.
3. The optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture according to claim 1, characterized in that, By adjusting the characteristic distance d between the imaging projection module and the light source array module and the beam splitting ratio of the beam splitter, the scattering angle θ of each diffraction order between the light source array module and the imaging projection module is adjusted. x and θ y Matching is performed, and then the displacement step size of the matrix A information relative to the matrix B in the x and y directions is adjusted.
4. The optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture according to claim 1, characterized in that, The light source array module and the signal modulation module are misaligned with different displacement step sizes in the two orthogonal directions x and y, so that the matrix A information partially overlaps the information of different rows or columns in the x direction, and then completely offsets the overlapped information in the y direction which is orthogonal to x.
5. The optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture according to claim 1, characterized in that, The system also includes any one or more of the following: - The light source array module adopts a light-emitting element array or a spatial light modulator; - The imaging projection module includes one or more Damman gratings, which are either one-dimensional or two-dimensional Damman gratings; and employs metasurface beam splitters or metamaterial beam splitters; - The signal modulation module uses a spatial light modulator; - The detection module includes a lens and a photodetector array. The lens is used to converge optical signals of different diffraction orders to different positions at specific diffraction angles to perform a summation operation. The photodetector array is used to detect and perform photoelectric conversion on the matrix C information of the matrix-matrix multiplication result.
6. The optical matrix-matrix multiplication calculation system based on a multi-imaging projection architecture according to any one of claims 1-5, characterized in that, Also includes: Electronic control components; The electronic control component includes hardware components and control components; wherein: The hardware components include: a precision displacement platform, a servo drive motor, a data acquisition card, and a clock synchronization device. The servo drive motor is used to control the movement step size of the precision displacement platform. The precision displacement platform is used to control the multi-dimensional precision adjustment of all functional modules in the optical matrix-matrix multiplication component. The data acquisition card is used to acquire the matrix information of the matrix-matrix multiplication result detected in the detection module. The clock synchronization device is used to realize the synchronous uploading and downloading of loaded information in the light source array module, signal modulation module, and detection module. The control components include: a LabVIEW driver module, a C language driver module, and a microcontroller control module. The LabVIEW driver module provides a graphical human-computer interaction interface, the C language driver module defines the operation instructions for the hardware components, and the microcontroller control module controls the hardware components according to the operation instructions defined by the C language driver module.
7. An optical matrix-matrix multiplication calculation method based on a multi-imaging projection architecture, characterized in that, include: The input optical signal is loaded with matrix A information to obtain an optical signal carrying matrix A information; Optical signals with different displacement step sizes and different diffraction angle spectra are generated in the two mutually orthogonal directions x and y for the optical signal carrying matrix A information. Load matrix B information, and multiply the optical signal carrying matrix A information with different displacement step size and different diffraction angle spectrum with matrix B information to obtain the multiplication information of corresponding elements of matrix A information and matrix B information at different positions, and then obtain the optical signal carrying the multiplication information. The optical signals carrying the multiplication information are converged to perform a summation operation on the multiplication information, thereby obtaining the multiplication information of matrix A and matrix B. The imaging projection module includes multiple beam splitters. These beam splitters generate several sub-beams in two mutually orthogonal directions (x and y) from the optical signal carrying matrix A information, forming different displacement step sizes and diffraction angle spectra. Each sub-beam is incident on a different position of the signal modulation module according to its respective diffraction angle, realizing parallel translation and replication operations of the optical signal carrying matrix A information. The matrix A information carried in each sub-beam incident on the signal modulation module is multiplied with matrix B information at the pixel level, realizing parallel multiplication operations between matrix A information and matrix B information. The parallel multiplication operation between matrix A and matrix B includes: performing a multiplication operation on matrix A and matrix B, and summing the multiplied information; wherein, the multiplication operation on matrix A and matrix B includes: The matrix A information is translated and copied in the x direction, and the movement step size is controlled to overlap the different rows or columns of matrix A to the same row or column. Then the overlapping part is truncated to obtain matrix A' containing the overlapping information. The matrix A' containing overlapping information is translated and copied in the y direction orthogonal to x, and the movement step size is controlled to make the matrix A' containing overlapping information completely offset in the y direction, thus obtaining A'' containing overlapping information; According to the matrix multiplication operation rules, the matrix A'' containing overlapping information is aligned and multiplied with the matrix B, and then the multiplication information of the matrix A'' and matrix B containing overlapping information is obtained at the corresponding positions; The multiplication information propagates along diffraction angle spectra in the x and y directions until the multiplication information is completely separated from each other; The summation operation on multiplied information includes: The completely separated sub-beams carrying information about the multiplication of the corresponding rows or columns of matrix A with the columns or rows of matrix B are converged to different positions on the detection surface of the detection module according to their respective diffraction angle spectra, thereby realizing the summation operation of the multiplication information.
8. The optical matrix-matrix multiplication calculation method based on a multi-imaging projection architecture according to claim 7, characterized in that, Optical signals of different diffraction orders in the x-direction at different angles θ x The propagation and control of the characteristic distance d cause the information from different rows or columns of matrix A to partially overlap in the same row or column, and optical signals of different diffraction orders in the y-direction are emitted at different angles θ. y The propagation and control of the characteristic distance d completely separate the matrix A information that overlaps in the same row or column in the y direction. The sub-beams containing the overlapping information are transmitted to different positions of matrix B according to their respective diffraction angle spectra. The optical signals of each diffraction order that are completely separated in the y direction carry all the information of the matrix A that partially overlaps in the same row or column.
9. The optical matrix-matrix multiplication calculation method based on a multi-imaging projection architecture according to claim 8, characterized in that, The elements in matrix A are multiplied in parallel with the corresponding elements in matrix B, including: Perform a multiplication operation between the elements in matrix A and the corresponding elements in matrix B: ; A mn ∙B nm =C nn ; Among them, [a 11 a 12 , ..., a nm ] represents an element of matrix A, [b 11 b 12 , ..., b nm ] represents the element of matrix B, and the subscripts n and m are the number of rows and columns of matrices A, B and C, respectively; The elements in matrix A of the multiplication information are multiplied by the corresponding elements in matrix B. The summation operation is then performed. c 11 = [a 11 , a 21 , …, a n1 ] × [b 11 , b 12 , …, b n1 ], c 12 = [a 11 , a 21 , …, a n1 ] ×[b 12 , b 22 , …, b n2 ], …; c 21 = [a 21 , a 22 , …, a n2 ] × [b 12 , b 22 , …, b n2 ], c 22 = [a 21 , a 22 , …, a n2 ] ×[b 12 , b 22 , …, b n2 ], …; …, c n1 = [a 1m , a 2m , …, a nm ] × [b 11 , b 12 , …, b n1 ], c n2 = [a 1m , a 2m , …, a nm ] ×[b 12 , b 22 , …, b n2 ], …。 10. The optical matrix-matrix multiplication calculation method based on a multi-imaging projection architecture according to any one of claims 7-9, characterized in that, The loaded matrix A information, the loaded matrix B information, and the obtained matrix-matrix multiplication result matrix C information are all analog quantities. The matrix-matrix multiplication result matrix C information is quantized into a digital result after digital processing to realize analog optical matrix-matrix operations.
11. The optical matrix-matrix multiplication calculation method based on a multi-imaging projection architecture according to any one of claims 7-9, characterized in that, The high-bit matrix A and high-bit matrix B to be processed are represented as multiple encoded low-bit matrices, respectively, to obtain encoded low-bit matrix A and low-bit matrix B. The low-bit matrix A and low-bit matrix B are used as the loaded matrix information to obtain the low-bit matrix-matrix multiplication result matrix information. The low-bit matrix-matrix multiplication result matrix information is decoded into high-bit matrix-matrix multiplication result matrix C information to realize digital optical matrix-matrix operations.
Citation Information
Patent Citations
Matrix vector multiplier based on photoelectric calculation array and operation method thereof
CN110263296A
Photon matrix vector multiplier based on wavelength routing and operation method thereof
CN113126693A
