Optical Convolution Computation System and Method Based on Multi-Imaging Projection Architecture

Abstract

This invention provides an optical convolution computation system and method based on a multi-imaging projection architecture, comprising: a light source array module, which loads convolution kernel matrix information onto the input optical signal to obtain an optical signal carrying the convolution kernel matrix information; an imaging projection module, which generates optical signals of different diffraction orders from the optical signal carrying the convolution kernel matrix information and outputs them to a signal modulation module; a signal modulation module, which loads the input matrix information and performs a multiplication operation between the convolution kernel matrix information and the input matrix information to obtain the multiplication information of corresponding elements at different positions of the two matrix information; and a detection module, which converges the optical signals carrying the multiplication information and performs a summation operation on the multiplication information to obtain the convolution result matrix information. This invention realizes a new technical route for large-scale, high-precision, and fully parallel optical computing, providing a general and efficient solution to meet the needs of artificial intelligence, neural network image processing, and other tasks for massive convolution operations.

Description

Technical Field

[0001] This invention relates to the field of optical computing technology, and more specifically, to an optical convolution computing system and method based on a multi-imaging projection architecture. Background Technology

[0002] With the rise of the third wave of artificial intelligence, represented by deep learning, multi-layered neural network structures, through training massive amounts of parameters, can extract specific pattern features from large datasets and have been widely applied in various complex scenarios, such as autonomous driving, scientific computing, trend prediction, and machine vision. However, approximately 80% of the computations in deep learning involve convolutional calculations between neural network layers, which places stringent demands on the computing power and power consumption of current serial computing hardware based on the von Neumann architecture. Although developing new AI hardware accelerators can accelerate AI algorithms, even GPUs, TPUs, FPGAs, or ASICs still consume significant power and time to process massive convolutional calculations. The number of operations required for convolutional calculations between two N×N matrices is approximately N. 4 The sheer scale of computations, especially with the continuous increase in neural networks, far exceeds the computing power of current AI hardware accelerator architectures. Therefore, there is an urgent need to develop new convolutional computing accelerators to meet the computational power and power consumption demands of the rapidly developing deep learning technology. Optical computing, due to its inherent parallelism, ultra-fast propagation speed, extremely low power consumption, and absence of electromagnetic interference, has experienced a resurgence and emerged as a highly promising solution for matrix computing in the post-Moore's Law era.

[0003] Currently, optical computing mainly falls into two mainstream approaches. One is a matrix-vector multiplier based on a planar optical waveguide architecture, primarily composed of cascaded Mach-Zehnder interferometers (Prior Technology 1: Nat. Photon. 11, 441 (2017)) or microring resonators (Prior Technology 2: Nature 569, 208 (2019), Prior Technology 3: Nature 589, 52 (2021), Prior Technology 4: Nature 589, 44 (2021)). This approach offers the best compatibility with current microelectronic processes and is a naturally integrated structure, thus attracting widespread attention, especially from industry. However, the 2D integrated optical waveguide architecture is limited by its two-dimensional expansion; both its input and output must be expanded into one-dimensional vectors, severely restricting its scalability. Furthermore, this one-dimensional expansion architecture does not fully utilize the two-dimensional optical interconnect capabilities, resulting in an insurmountable upper limit to computational power.

[0004] Furthermore, a spatial diffractive optical neural network based on diffractive optical elements (Prior Art 5: Science 361, 1004-1008 (2018)) has been proposed. This system uses 3D printing technology to cascade trained diffractive optical layers together to achieve the function of a convolutional neural network. This approach uses cascaded diffractive optical devices to achieve spatial interconnection between different convolutional layers, fundamentally avoiding the obstacle of achieving high-density interconnection with one-dimensional vectors, and fully utilizing the optical spatial interconnection capability. Essentially, this multi-layered cascaded diffractive optical element is a three-dimensional subwavelength optical element capable of achieving precise modulation of complex electromagnetic fields. However, the practical implementation of such a subwavelength three-dimensional optical device faces significant challenges. Moreover, once the diffractive optical element is manufactured, the diffractive optical network structure cannot be dynamically adjusted.

[0005] Recently, another convolution calculation system based on the 4F regular signal system has been proposed (Prior Art 6: Optica7,1812-1819(2020)). This technology realizes the function of convolutional neural network by placing two amplitude-modulated DMDs on the object plane and image plane of the 4F system respectively. However, due to the limitation of the Fourier transform relationship between the object plane and image plane of the 4F system, it is difficult to achieve large-scale and high-precision convolution calculation.

[0006] In fact, space optical computing systems, due to their inherent two-dimensional parallel characteristics, were extensively studied in the 1980s and 1990s, resulting in various signal processing systems with practical value. In 1992, Zhou Changhe et al. implemented a binary optical matrix-vector multiplier based on the shadow projection method (Prior Art 7: Opt. Lett. 17, 1800-1802 (1992)). However, in the traditional shadow projection method, the information of matrix A is directly projected onto the plane containing matrix B to achieve the multiplication of matrices A and B. There is a characteristic distance d0 = Δ / (tanθp) between the planes containing matrices A and B, where Δ is the actual physical size of the pixels in the planes of matrices A and B, θp is the angle between light of different diffraction orders and the z-axis, p = ±1, ±2, ... Therefore, diffraction interference introduced by the propagation of the light field is inevitable. This means that the smaller the matrix pixels, the more obvious the diffraction effect and the worse the calculation accuracy. While increasing the size of the matrix pixels can further reduce computational errors, and larger pixel sizes can also reduce alignment errors among all pixels in the planes containing matrices A and B, while maintaining a sufficiently high signal-to-noise ratio, the larger the matrix unit, the larger the distance d, and the more significant the diffraction effect becomes. Therefore, there is a severe constraint between the computational power and accuracy of the traditional shadow delivery method's optical computing architecture, making large-scale convolution impossible. This severely limits the computational power and accuracy of this scheme.

[0007] In summary, planar integrated optical waveguide solutions have a clear computational ceiling. Although spatial diffractive optics solutions can theoretically achieve large-scale matrix calculations, in practice, how to realize three-dimensional subwavelength optical elements with precise control of complex electromagnetic fields, especially in the optical band, still faces insurmountable difficulties. Summary of the Invention

[0008] To address the aforementioned shortcomings in the prior art, this invention provides an optical convolution calculation system and method based on a multi-imaging projection architecture.

[0009] According to one aspect of the present invention, an optical convolution computing system based on a multi-imaging projection architecture is provided, comprising: an optical convolution component, the optical convolution component including: a light source array module, an imaging projection module disposed at the rear end of the light source array module, a signal modulation module disposed at the rear end of the imaging projection module, and a detection module disposed at the rear end of the signal modulation module; wherein:

[0010] The light source array module is used to load convolution kernel matrix information onto the input optical signal to obtain an optical signal carrying convolution kernel matrix information;

[0011] The imaging projection module is used to generate optical signals of different diffraction orders from the optical signal carrying the convolution kernel matrix information and output them to the signal modulation module.

[0012] The signal modulation module is used to load input matrix information and multiply the convolution kernel matrix information carried by the optical signals of different diffraction orders with the input matrix information to obtain the multiplication information of corresponding elements of the convolution kernel matrix information and the input matrix information at different positions, thereby obtaining an optical signal carrying the multiplication information.

[0013] The detection module is used to converge the optical signal carrying the multiplication information, perform a summation operation on the multiplication information, and obtain the convolution result matrix information.

[0014] Preferably, the light source array module uses one or more optical signals with different characteristics to simultaneously load one or more convolution kernel matrix information. After the optical signals with different characteristics carrying one or more convolution kernel matrix information pass through the imaging projection module and the signal modulation module, they are multiplied with the input matrix information respectively. Finally, they are converged by the detection module to detect the convolution result matrix information of one or more convolution kernel matrix information and the input matrix information.

[0015] Preferably, the imaging projection module includes multiple beam splitters, which divide the optical signal carrying convolution kernel matrix information into several sub-beams. Each sub-beam is incident on a different position of the signal modulation module according to its own diffraction angle, thereby realizing parallel translation of the optical signal carrying convolution kernel matrix information. The convolution kernel matrix information carried in each sub-beam incident on the signal modulation module is multiplied by the input matrix information at the pixel level, thereby realizing parallel multiplication of the convolution kernel matrix information and the input matrix information.

[0016] Preferably, the parallel multiplication operation between the convolution kernel matrix information and the input matrix information includes:

[0017] Multiply the elements in the convolution kernel matrix information with the corresponding elements in the input matrix information;

[0018] After sliding the elements in the convolution kernel matrix information with a fixed stride through the input matrix information, the above multiplication steps are repeated.

[0019] After traversing all positions, the multiplication information of the convolution kernel matrix information and the corresponding elements of the input matrix information at different positions is obtained;

[0020] The summation operation on the multiplied information includes:

[0021] The optical signal carrying the multiplication information is focused onto different positions on the detection surface of the detection module according to the tilt component of the diffraction order, so as to realize the summation operation of the multiplication information.

[0022] Preferably, the plane where the light source array module is located and the plane where the signal modulation module is located satisfy the object-image conjugate relationship.

[0023] Preferably, by adjusting the feature distance between the imaging projection module and the light source array module, the scattering angles of each diffraction order of the light source array module and the imaging projection module are matched, thereby changing the sliding step size of the convolution kernel matrix information on the input matrix information.

[0024] Preferably, the light source array module and the signal modulation module are misaligned in the lateral position, so that the convolution kernel matrix information and the input matrix information are misaligned.

[0025] Preferably, the light source array module is an array of light-emitting elements or a spatial light modulator.

[0026] Preferably, the imaging projection module includes one or more Damman gratings, wherein the Damman gratings are one-dimensional Damman gratings or two-dimensional Damman gratings.

[0027] Preferably, the imaging projection module is a metasurface beam splitter or a metamaterial beam splitter.

[0028] Preferably, the signal modulation module employs a spatial light modulator.

[0029] Preferably, the detection module includes a lens and an optical receiver array, wherein the lens is used to converge optical signals of different diffraction orders to different positions according to a specific diffraction angle to perform a summation operation, and the optical receiver array is used to detect the information of the convolution result matrix.

[0030] Preferably, the system further includes: electronic control components;

[0031] The electronic control component includes hardware components and control components; wherein:

[0032] The hardware components include: a precision displacement platform, a servo drive motor, a data acquisition card, and a phase-locked synchronization device. The servo drive motor is used to control the movement step size of the precision displacement platform. The precision displacement platform is used to control the multi-dimensional precision adjustment of all functional modules in the optical convolution component. The data acquisition card is used to acquire the convolution result matrix information detected by the detection module. The phase-locked synchronization device is used to realize the synchronous uploading and downloading of loaded information in the light source array module, signal modulation module, and detection module.

[0033] The control components include: a LabVIEW driver module, a C language driver module, and a microcontroller control module. The LabVIEW driver module provides a graphical human-computer interaction interface, the C language driver module defines the operation instructions for the hardware components, and the microcontroller control module controls the hardware components according to the operation instructions defined by the C language driver module.

[0034] According to a second aspect of the present invention, an optical convolution calculation method based on a multi-imaging projection architecture is provided, comprising:

[0035] The input optical signal is loaded with convolution kernel matrix information to obtain an optical signal carrying convolution kernel matrix information;

[0036] Optical signals of different diffraction orders are generated from the optical signal carrying the convolution kernel matrix information;

[0037] The input matrix information is loaded, and the convolution kernel matrix information carried by the optical signals of different diffraction orders is multiplied with the input matrix information to obtain the multiplication information of corresponding elements of the convolution kernel matrix information and the input matrix information at different positions, thereby obtaining the optical signal carrying the multiplication information;

[0038] The optical signals carrying the multiplication information are converged to perform a summation operation on the multiplication information, thereby obtaining the convolution result matrix information.

[0039] Preferably, the optical signals of different diffraction orders are transmitted to different positions of the input matrix at different angles, wherein the optical signal of each diffraction order carries all the information of the convolution kernel matrix.

[0040] Preferably, the multiplication operation between the convolution kernel matrix information and the input matrix information includes:

[0041] Multiply the elements in the convolution kernel matrix information with the corresponding elements in the input matrix information;

[0042] After sliding the elements in the convolution kernel matrix information with a fixed stride through the input matrix information, the above multiplication steps are repeated.

[0043] After traversing all positions, the multiplication information of the corresponding elements of the convolution kernel matrix information and the input matrix information at different positions is obtained.

[0044] Preferably, the summation operation on the multiplied information includes:

[0045] The optical signal carrying the multiplication information is focused onto different positions on the detection surface of the detection module according to the tilt component of the diffraction order, so as to realize the summation operation of the multiplication information.

[0046] Preferably, multiplying the elements in the convolution kernel matrix information with the corresponding elements in the input matrix information includes:

[0047] Perform a multiplication operation between the elements in the convolution kernel matrix information and the corresponding elements in the input matrix information:

[0048] {[a 11 *b jk a 12 *b j(k+1) , ..., a 1n *b j(k+n-1) ];

[0049] [a 21 *b (j+1)k ,a 22 *b (j+1)(k+1) , ..., a 2n *b (j+1)(k+n-1) ];

[0050] …

[0051] [a m1 *b (j+m-1)k a m2*b (j+m-1)(k+1) , ..., a mn *b (j+m-1)(k+n-1) ]};

[0052] Among them, a 11 a 12 , ..., a mn Let b be an element of the input matrix. 11 b 12 , ..., b mn The elements are the convolution kernel matrix, with subscripts m and n being the row and column numbers of the kernel matrix, respectively, and subscripts j and k being the row and column numbers of the overlapping elements of the input matrix and the kernel matrix, respectively.

[0053] Preferably, the elements in the convolution kernel matrix information of the multiplication information are summed with the corresponding elements in the input matrix information:

[0054] {c 11 =[a 11 *b 11 +a 12 *b 12 +…+a 1n *b 1n ]+[a 21 *b 21 +a 22 *b 22 +…+a 2n *b 2n ]+…+[a m1 *b m1 +a m2 *b m2

[0055] +…+a mn *b mn ];

[0056] c 1s =[a 1(1+s) *b 1(1+s) +a 1(2+s) *b 1(2+s) +…+a 1(n+s) *b 1(n+s) ]+[a 2(1+s) *b 2(1+s) +a 2(2+s) *b 2(2+s) +…+a 2(n+s)

[0057] *b 2(n+s) ]+…+[a m(1+s) *b m(1+s) +a m(2+s) *b m(2+s) +…+am(n+s) *b m(n+s) ];

[0058] …

[0059] c ts =[a (1+t)(1+s) *b (1+t)(1+s) +a (1+t)(2+s) *b (1+t)(2+s) +…+a (1+t)(n+s) *b (1+t)(n+s) ]+[a (2+t)(1+s) *b (2+t)(1+s) +a (2+t)(2+s) *b (2+t)(2+s) +…+a (2+t)(n+s) *b (2+t)(n+s) ]+…+[a (m+t)(1+s) *b (m+t)(1+s) +a (m+t)(2+s) *b (m+t)(2+s) +…+a (m+t)(n

[0060] +s) *b (m+t)(n+s) ]};

[0061] Where the subscripts t and s represent the lengths of the horizontal and vertical sliding steps of the convolution kernel matrix on the input matrix, respectively.

[0062] Preferably, the loaded convolution kernel matrix information, the loaded input matrix information, and the obtained convolution result matrix information are all analog quantities. The convolution result matrix information is quantized into a digital result after digital processing to realize analog convolution operation.

[0063] Preferably, the high-bit convolution kernel matrix information and the high-bit input matrix information to be processed are represented as multiple encoded low-bit matrices, respectively, to obtain encoded low-bit convolution kernel matrix information and low-bit input matrix information; the low-bit convolution kernel matrix information and the low-bit input matrix information are used as the loaded matrix information to obtain low-bit convolution result matrix information; the low-bit convolution result matrix information is decoded into high-bit convolution result matrix information to realize digital convolution operation.

[0064] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0065] The optical convolution computation system and method based on a multi-imaging projection architecture provided by this invention can perform convolution operations on the kernel matrix B and the input matrix A in a highly parallel, high-speed, and low-power manner, and directly obtain the convolution result matrix C on the end face of the detection module (detector). Based on the system and method provided by this invention, the convolution of arbitrary bit matrices with large-scale parallelism and sufficiently high accuracy can be efficiently calculated. Moreover, the convolution is universal, and the obtained computation results are very easy to port to any other computing platform. By developing a high-speed spatial light modulator with higher contrast, optimizing a dedicated projection imaging system, and configuring a dedicated dot matrix light source, an optical convolution processor with higher computing power and lower energy consumption compared to an electronic computer can be constructed. In addition, due to the characteristics of the imaging system itself, by cascading multiple 4F systems and employing additional multiplexing degrees of freedom, the computing power of the system can be multiplied, which is expected to construct a hybrid optoelectronic computer center or data center based on an optical convolutioner of a multi-imaging projection architecture.

[0066] The optical convolution computation system and method based on a multi-imaging projection architecture provided by this invention significantly improves the pixel utilization of the spatial light modulator in the optical convolution computation system, reduces the requirements for the dynamic detection range of the detection module, and can further improve the computational accuracy of the optical convolution system. Compared with electronic AI accelerators and other optical computing solutions, the system and method provided by this invention can realize general-purpose, arbitrary-base digital optical convolution computation, and has the characteristics of high speed, low power consumption, high parallelism, high tolerance, large scale, and reconfigurability. The system and method provided by this invention lays the research foundation for digital optical computing and is expected to further develop digital optical computing systems based on matrix transformation, decomposition, and other operations. It has significant application value and good economic benefits in deep learning and other fields involving a large number of matrix operations.

[0067] The optical convolution calculation system and method based on a multi-imaging projection architecture provided by this invention is an optical convolution calculation system with truly large-scale parallelism and sufficiently high precision. After loading the convolution kernel matrix B and the input matrix A into the input module, the large-scale, high-precision optical convolution calculation result matrix C can be directly obtained in the detection module after the optical signal passes through the system once; wherein, the matrix...

[0068] The planes containing A and B satisfy the object-image conjugate relationship, avoiding the diffraction problem present in traditional shadow projection methods. By introducing an imaging projection module, each diffraction order sub-beam carrying all the information of the convolution kernel matrix B is perfectly imaged onto the corresponding position in the plane of matrix A, realizing parallel multiplication operations. Parallel summation operations are achieved through lenses. Furthermore, by adjusting the distance d between matrix A and the imaging projection module and the diffraction angle θ of the beam splitter... pA perfect match is achieved to modify the parameters for convolution calculations. The system and method provided by this invention solve the problem of large-scale convolution operations required in practical AI applications such as image processing, machine vision, and autonomous driving, represented by convolutional neural networks, and have significant economic implications and development prospects. Attached Figure Description

[0069] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0070] Figure 1 This is a schematic diagram illustrating the working principle of an optical convolution calculation system based on a multi-imaging projection architecture in a preferred embodiment of the present invention. In this diagram, 101 is a light source array module, 102 is an imaging projection module, 103a is a Fourier transform lens L1, 103b is a Fourier transform lens L2, 104 is the focal plane of the Fourier transform lens L1, 105 is a signal modulation module, 106 is a detection module, 106a is a focusing lens L3, and 106b is an optical receiver array.

[0071] Figure 2 This is a schematic diagram of the architecture of an optical convolution calculation system based on a multi-imaging projection architecture according to an embodiment of the present invention. Wherein, 21 is the optical convolution part, 22 is the electronic control part, 2101 is the light source array module, 2102 is the imaging projection module, 2103 is the signal modulation module, 2104 is the detection module, 2201 is the low-bit convolution kernel matrix B loaded into the light source array module, 2202 is the low-bit input matrix A loaded into the signal modulation module, 2203 is the low-bit convolution result matrix C detected by the detection module, 2204 is the FPGA / CPU control unit, and 2205 is the signal synchronization lock-in amplifier.

[0072] Figure 3 This is a schematic diagram of an optical convolution calculation system based on a multi-imaging projection architecture in a specific application example of the present invention. The optical elements are arranged in the following order: 301 is a light source array, 302 is a Damman grating, 303 is a first Fourier transform lens, 304 is a second Fourier transform lens, 305 is a spatial light modulator, 306 is a focusing lens, and 307 is a detector.

[0073] Figure 4 This is a schematic diagram of the convolution experiment results of a 10×10 1-bit randomly generated matrix in a specific application example of the present invention. Among them, (a) is the convolution result detected on the detector sCMOS1, (b) is the conversion of the convolution result on the detector sCMOS1 into the corresponding gray value, (c) is the theoretical value of the convolution result, and (d) is the absolute error between the theoretical value and the experimental value of the convolution result, that is, the theoretical value minus the experimental value.

[0074] Figure 5 This is a schematic diagram of the convolution experiment results of two 10×10 1-bit matrices in a specific application example of the present invention. (a) is the convolution result detected on the detector; (b) is the conversion of the convolution result on the detector into the corresponding grayscale value; (c) is the theoretical value of the convolution result; (d) is the error between the theoretical and experimental values ​​of the convolution; (e-1) is the decoding result of the theoretical value of the convolution result; and (e-2) is the decoding result of the experimental value of the convolution result.

[0075] Figure 6 This is a schematic diagram illustrating the convolutional experimental results of two 10×10 1-bit matrices based on a hybrid positive and negative encoding with arbitrary bases in a specific application example of the present invention. In the diagram, (a) is the convolutional result detected by the detector, (b) is the conversion of the convolutional result on the detector into the corresponding grayscale value, (c) is the theoretical value of the convolutional result, (d) is the error between the theoretical and experimental values ​​of the convolution, (e-1) is the decoding result of the theoretical value of the convolutional result, and (e-2) is the decoding result of the experimental value of the convolutional result.

[0076] Figure 7 This is a schematic diagram of the experimental results of convolution of a 20×20 1-bit matrix based on arbitrary base encoding in a specific application example of the present invention. Among them, (a) is the theoretical convolution value, (b) is the light intensity distribution, (c) is the experimental convolution value, and (d) is the error distribution of the two 20×20 1-bit matrices.

[0077] Figure 8 This is a schematic diagram of the experimental results of convolution of a 10×10 8-bit matrix based on arbitrary base encoding in a specific application example of the present invention. Among them, (a) is the theoretical full convolution value, (b) is the light intensity distribution, (c) is the experimental full convolution value, and (d) is the error distribution of the two 10×10 8-bit matrices.

[0078] Figure 9 This is a schematic diagram illustrating the experimental results of 180×224 8-bit matrix convolution based on arbitrary base encoding in a specific application example of this invention. (a) shows the theoretical convolution result, (b) shows the experimental convolution result, (c) shows the experimental probe spot distribution, (d) shows a magnified view of a portion of the experimental probe spot (the red cross indicates the calculated centroid position), (e) shows the error distribution compared to the maximum value of 1 / 256, where green circles indicate that the error corresponding to that convolution element value is less than 1 / 256 of the maximum value of the theoretical convolution result, and (f) shows the percentage of the convolution matrix error distribution. The above experimental results demonstrate that the optical convolution system based on a multi-imaging projection architecture proposed in this invention has excellent robustness and is expected to achieve larger-scale and higher-precision optical convolution calculations.

[0079] Figure 10This is a schematic diagram of the experimental results of implementing a convolutional neural network based on the MNIST dataset in a specific application example of the present invention. Among them, (a) is the convolution operation performed after encoding the convolution kernel as positive and negative convolution kernels, (b) is the theoretical result of the error convolution calculation between the experimental result and the error of the number 7, (c) is the confusion matrix of the optical convolution system based on the blind test results of 1000 images, and (d) is the confusion matrix of the computer based on the blind test results of 1000 images.

[0080] Figure 11 This is a flowchart of an optical convolution calculation method based on a multi-imaging projection architecture in one embodiment of the present invention. Detailed Implementation

[0081] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0082] One embodiment of the present invention provides an optical convolution computing system based on a multi-imaging projection architecture, comprising: an optical convolution component, the optical convolution component including: a light source array module, an imaging projection module disposed at the rear end of the light source array module, a signal modulation module disposed at the rear end of the imaging projection module, and a detection module disposed at the rear end of the signal modulation module; wherein:

[0083] The light source array module is used to load convolution kernel matrix information onto the input optical signal to obtain an optical signal carrying the convolution kernel matrix information;

[0084] The imaging projection module is used to generate optical signals of different diffraction orders from the optical signal carrying the convolution kernel matrix information and output them to the signal modulation module;

[0085] The signal modulation module is used to load the input matrix information and multiply the convolution kernel matrix information carried by the optical signals of different diffraction orders with the input matrix information to obtain the multiplication information of the corresponding elements of the convolution kernel matrix information and the input matrix information at different positions, thereby obtaining the optical signal carrying the multiplication information.

[0086] The detection module is used to converge the optical signals carrying multiplication information, perform the summation operation on the multiplication information, and obtain the convolution result matrix information.

[0087] In this embodiment, the multiplication operation performed by the signal modulation module and the summation operation performed by the detection module together complete the convolution operation of the two matrices to be processed.

[0088] In a preferred embodiment, the light source array module uses one or more optical signals with different characteristics to simultaneously load one or more convolution kernel matrix information. After the optical signals with different characteristics carrying one or more convolution kernel matrix information pass through the imaging projection module and the signal modulation module, they are multiplied with the input matrix information respectively. Finally, they are converged by the detection module to detect the convolution result matrix information of one or more convolution kernel matrix information and the input matrix information.

[0089] Furthermore, optical signals possess various characteristics, including but not limited to wavelength, mode, and polarization.

[0090] In a preferred embodiment, the imaging projection module includes multiple beam splitters, which divide the optical signal carrying convolution kernel matrix information into several sub-beams. Each sub-beam is incident on a different position of the signal modulation module according to its own diffraction angle, thereby realizing parallel translation of the optical signal carrying the convolution kernel matrix information. The convolution kernel matrix information carried in each sub-beam incident on the signal modulation module is multiplied by the input matrix information at the pixel level, thereby realizing parallel multiplication of the convolution kernel matrix information and the input matrix information.

[0091] As a preferred embodiment, the parallel multiplication operation of the convolution kernel matrix information and the input matrix information includes:

[0092] Multiply the elements in the convolution kernel matrix information with the corresponding elements in the input matrix information;

[0093] After sliding the elements of the convolution kernel matrix information through the input matrix information with a fixed stride, the above multiplication steps are repeated.

[0094] After traversing all positions, we obtain the element-wise multiplication information of the convolution kernel matrix and the input matrix at different positions;

[0095] The summation operation on multiplied information includes:

[0096] The optical signals carrying multiplication information are focused onto different positions on the detection surface of the detection module according to the tilt components of the diffraction order, so as to realize the summation operation of the multiplication information.

[0097] In this preferred embodiment, sliding the elements of the convolution kernel matrix information in the input matrix information with a fixed step size means that after passing through the imaging projection module, the optical signal is divided into multiple sub-beams carrying the convolution kernel matrix information. Different sub-beams propagate to different positions of the signal modulation module according to specific diffraction directions, which realizes the sliding of the convolution kernel matrix information in the input matrix information. This operation is equivalent to mathematically translating and sliding the convolution kernel.

[0098] In a preferred embodiment, the plane where the light source array module is located and the plane where the signal modulation module is located satisfy the object-image conjugate relationship.

[0099] In a preferred embodiment, by adjusting the feature distance between the imaging projection module and the light source array module, the scattering angles of each diffraction order of the light source array module and the imaging projection module are matched, thereby changing the sliding step size of the convolution kernel matrix information on the input matrix information.

[0100] In a preferred embodiment, the light source array module and the signal modulation module are misaligned in the lateral position, so that the convolution kernel matrix information and the input matrix information are misaligned.

[0101] As a preferred embodiment, the light source array module employs an array of light-emitting elements or a spatial light modulator.

[0102] In a preferred embodiment, the imaging projection module includes one or more Damman gratings, which are either one-dimensional or two-dimensional Damman gratings.

[0103] As a preferred embodiment, the imaging projection module uses a metasurface beam splitter or a metamaterial beam splitter.

[0104] As a preferred embodiment, the signal modulation module employs a spatial light modulator.

[0105] In a preferred embodiment, the detection module includes a lens and an optical receiver array. The lens is used to converge optical signals of different diffraction orders to different positions at specific diffraction angles to perform a summation operation, and the optical receiver array is used to detect the information of the convolution result matrix.

[0106] As a preferred embodiment, the system further includes: electronic control components;

[0107] Electronic control components include hardware components and control components; among which:

[0108] The hardware components include: a precision displacement platform, a servo drive motor, a data acquisition card, and a phase-locked synchronization device. The servo drive motor is used to control the movement step size of the precision displacement platform. The precision displacement platform is used to control the multi-dimensional precision adjustment of all functional modules in the optical convolution component. The data acquisition card is used to acquire the convolution result matrix information detected in the detection module. The phase-locked synchronization device is used to realize the synchronous uploading and downloading of the loaded information in the light source array module, signal modulation module, and detection module.

[0109] The control components include: a LabVIEW driver module, a C language driver module, and a microcontroller control module. The LabVIEW driver module provides a graphical human-computer interaction interface, the C language driver module defines the operation instructions for the hardware components, and the microcontroller control module controls the hardware components according to the operation instructions defined by the C language driver module.

[0110] The optical convolution computing system based on a multi-imaging projection architecture provided in this embodiment can be optimized and developed based on any form of optical system that satisfies the object-image conjugate relationship between the convolution kernel matrix and the input matrix.

[0111] A preferred embodiment of the present invention provides an optical convolution computation system based on a multi-imaging projection architecture, which can achieve... High-precision, large-scale convolution computation, where A is the input matrix, B is the convolution kernel matrix, and C is the convolution result matrix. This represents a convolution operation, which involves multiplying and summing the elements of the convolution kernel matrix B with the corresponding elements of the input matrix A. The kernel matrix B slides across the input matrix A with a fixed stride, repeatedly performing these multiplication and summation operations until all positions are traversed, resulting in the convolution matrix. The output convolution matrix C can be a fully convolutional matrix or a matrix of the same size as the input matrix A.

[0112] The optical convolution computing system based on a multi-imaging projection architecture provided in this preferred embodiment includes two parts: an optical convolution part and an electronic control part. The optical convolution part includes a light source array module, an imaging projection module, a signal modulation module, and a detection module; the electronic control part includes various hardware components and control components.

[0113] In a preferred embodiment, the imaging projection module employs a large-scale beam splitter to achieve parallel copying, translation, and multiplication operations of the convolution kernel matrix B on the input matrix A. The optical signal carrying information about the convolution kernel matrix B is divided into several sub-beams after passing through the large-scale beam splitter. Each sub-beam undergoes a parallel copying operation with matrix A. Each sub-beam is incident on different positions of the signal modulation module carrying the matrix A information according to its respective diffraction angle; this corresponds to a parallel translation operation. The optical signals incident on different positions of the signal modulation module are then subjected to pixel-level multiplication of the kernel matrix B and matrix A by the signal modulation module; this corresponds to a parallel multiplication operation. The above process can be represented as follows:

[0114] {[a 11 *b jk a 12 *b j(k+1) , ..., a 1n *b j(k+n-1) ];

[0115] [a 21 *b (j+1)k ,a 22 *b (j+1)(k+1) , ..., a 2n *b (j+1)(k+n-1) ];

[0116] …

[0117] [a m1 *b (j+m-1)k a m2 *b (j+m-1)(k+1) , ..., a mn *b (j+m-1)(k+n-1) ]}.

[0118] Among them, a 11 a 12 , ..., a mn b are elements of the input matrix A. 11 b 12 , ..., b mn These are elements of the convolution kernel matrix B. The subscripts m and n are the row and column numbers of the convolution kernel matrix B, respectively, and the subscripts j and k are the row and column numbers of the overlapping part of the input matrix A and the convolution kernel matrix B.

[0119] The detection module uses lenses to perform the summation operation of the multiplied elements at corresponding positions in matrices A and B, that is:

[0120] {c 11 =[a 11 *b 11 +a 12 *b 12 +…+a 1n *b 1n ]+[a 21 *b 21 +a 22 *b 22 +…+a 2n *b 2n ]+…+[a m1 *b m1 +a m2 *b m2 +…+a mn *b mn ];

[0121] c 1s =[a 1(1+s) *b 1(1+s) +a 1(2+s) *b 1(2+s) +…+a 1(n+s) *b 1(n+s) ]+[a 2(1+s) *b 2(1+s) +a 2(2+s)*b 2(2+s) +…+a 2(n+s) *b 2(n+s) ]+…+[a m(1+s) *b m(1+s) +a m(2+s) *b m(2+s) +…+a m(n+s) *b m(n+s) ];

[0122] …

[0123] c ts =[a (1+t)(1+s) *b (1+t)(1+s) +a (1+t)(2+s) *b (1+t)(2+s) +…+a (1+t)(n+s) *b (1+t)(n+s) ]+[a (2+t)(1+s) *b (2+t)(1+s) +a (2+t)(2+s) *b (2+t)(2+s) +…+a (2+t)(n+s) *b (2+t)(n+s) ]+…+[a (m+t)(1+s) *b (m+t)(1+s) +a (m+t)(2+s) *b (m+t)(2+s) +…+a (m+t)(n+s) *b (m+t)(n+s) ]}.

[0124] Wherein, the subscripts m and n are the number of rows and columns of the convolution kernel matrix B, respectively, and the subscripts t and s are the lengths of the horizontal and vertical sliding steps of the convolution kernel matrix B on the input matrix A, respectively.

[0125] In a preferred embodiment, the light source array module is placed before the imaging projection module, and the light source array module is used to generate an optical signal that loads information of the convolution kernel matrix B.

[0126] The imaging projection module is positioned after the light source array module and before the signal modulation module. The imaging projection module mainly consists of a large-scale beam splitter and a projection lens group. The large-scale beam splitter is used to achieve parallel translation and replication of the optical signal carrying the information of the convolution kernel matrix B on the signal modulation module. The optical signals of different diffraction orders after passing through the large-scale beam splitter propagate at different angles θ. p The optical signals of different diffraction orders are transmitted to different positions on the plane of the signal modulation module, where p is the index of the diffraction order. Each diffraction order optical signal carries all the information of the convolution kernel matrix B and multiplies it with the information of the corresponding position of the matrix A loaded on the signal modulation module. The optical signals of different diffraction orders of the large-scale beam splitter are propagated to different positions on the plane of the signal modulation module, which corresponds to the parallel translation, sliding and multiplication operations of the convolution kernel matrix B on the input matrix A in the convolution calculation.

[0127] The signal modulation module is placed after the imaging projection module and before the detection module. The signal modulation module is used to modulate the optical signals of different diffraction orders that carry the information of the convolution kernel matrix B after passing through the imaging projection module, so as to realize the multiplication operation between the convolution kernel B and the input matrix A. The signal modulation module loads the information of the input matrix A.

[0128] The detection module is placed after the signal modulation module. The detection module is used to realize the convergence summation and detection of the convolution result. The optical signal carrying the corresponding element multiplication information of the B matrix and the A matrix at different positions is converged to different positions of the detection surface of the detection module according to the tilt component of the diffraction order to realize the summation operation. The lens element in the detection module performs the convergence operation on the optical signal carrying the A and B multiplication information, which corresponds to the summation operation of the corresponding element multiplication information of the convolution kernel matrix B and the input matrix A in the convolution calculation.

[0129] The optical convolution computation system based on a multi-imaging projection architecture provided in this preferred embodiment can perform both analog and digital convolution operations; wherein:

[0130] When performing analog convolution operations through the optical convolution computing system, the input matrix A and the convolution kernel matrix B are both analog quantities, and the convolution result matrix C obtained on the detection module is also an analog quantity. After digital processing, the convolution result matrix C is quantized into a digital result.

[0131] When performing high-precision, large-scale digital optical convolution operations through an optical convolution computing system, the high-bit matrix to be processed is first represented as multiple encoded low-bit matrices according to matrix encoding and decoding algorithms. Then, the encoded low-bit convolution kernel matrix B and the low-bit input matrix A are loaded onto the light source array module and the signal modulation module, respectively. When light passes through the system once, the low-bit convolution result matrix C is obtained on the detection surface of the detection module. Finally, the low-bit convolution result matrix is ​​decoded into a high-bit matrix.

[0132] In a preferred embodiment, the electronic control section includes hardware components and control components. The hardware components include a precision displacement platform, a servo drive motor, a data acquisition card, and a phase-locked loop (PLL) synchronization device. The servo drive motor controls the movement step size of the precision displacement platform, which controls the multi-dimensional precision adjustment of all optical elements in the optical convolution section. The data acquisition card is used to acquire the convolution result matrix detected by the detection module at high speed. The PLL synchronization device is used to synchronously upload and download the loaded information in the light source array module, signal modulation module, and detection module. The control components include a LabVIEW driver module, a C language driver module, and a microcontroller control module. The LabVIEW driver module is used to complete the graphical human-computer interaction interface, the C language driver module is used to define the operation instructions for the hardware components, and the microcontroller control module completes the control of all hardware components according to the operation instructions defined by the C language driver module.

[0133] As a preferred embodiment, the light source array module can be an array of light-emitting elements, including but not limited to LEDs, LDs, fiber arrays, vertical cavity surface-mount semiconductor laser arrays (VCSELs), or spatial light modulators (SLMs), including but not limited to liquid crystal spatial light modulators (LCSLMs), digital micromirror arrays (DMDs), microelectromechanical systems (MEMS), fiber arrays, and optical waveguide arrays.

[0134] As a preferred embodiment, the large-scale beam splitter can be a one-dimensional Damman grating with a beam splitting ratio of 1×3 to 1×128, or a two-dimensional Damman grating with a beam splitting ratio of 3×3 to 128×128 or larger. By combining Damman gratings, higher diffraction efficiency and larger-scale beam splitting effect can be achieved. Other beam splitting elements can also be used, including but not limited to diffractive optical elements (DOE), low-dimensional functional materials, metasurfaces, and metamaterials.

[0135] As a preferred embodiment, the signal modulation module includes, but is not limited to, various spatial light modulators (SLMs), such as liquid crystal spatial light modulators (LDSLMs), digital micromirror arrays (DMDs), microelectromechanical systems (MEMS), fiber optic arrays, and optical waveguide arrays.

[0136] In a preferred embodiment, the detection module includes a lens and an optical receiver array. The lens is used to converge and sum optical signals of different diffraction orders at different locations, and the optical receiver array is used to detect optical convolution signals, including but not limited to CMOS, CCD and various photodetector arrays.

[0137] The optical convolution computing system based on a multi-imaging projection architecture provided in this preferred embodiment can be optimized and developed based on any form of optical system that satisfies the object-image conjugate relationship between the convolution kernel matrix B and the input matrix A. It can be a transmissive optical system, a refractive optical system, or a reflective optical system. It can be a paraxial optical system, an off-axis optical system, or a planar space optical waveguide system. It includes, but is not limited to, various variations and combinations of single or multiple cascaded 4F canonical signal systems, microscopic systems, telescope systems, projection systems, and the above optical systems.

[0138] The optical convolution computing system based on a multi-imaging projection architecture provided in this preferred embodiment can further improve computing power through novel optical communication technologies that expand capacity, including but not limited to wavelength division multiplexing, mode division multiplexing, and polarization multiplexing.

[0139] The detection module is located after the signal modulation module. It is used to achieve convergence summation and detection of the convolution results. Through optical elements such as lenses in the detection module, the optical signals containing the element-wise multiplication information of the convolution kernel matrix B and the input matrix A at different positions are converged to different positions on the detection surface of the detection module according to the diffraction order to achieve the summation operation. The lens elements in the detection module converge the optical signals carrying the A and B multiplication information, corresponding to the summation operation of the element-wise multiplication information of the convolution kernel matrix B and the input matrix A in the convolution calculation. The detection module includes lenses and an optical receiver array. The lenses are used to achieve convergence summation of optical signals of different diffraction orders at different positions, and the optical receiver array is used to detect the optical convolution signals, including but not limited to CMOS and CCD detectors.

[0140] Compared to traditional shadow delivery methods, the optical convolution computing system based on a multi-imaging projection architecture provided in the above embodiments of the present invention effectively utilizes diffraction effects to achieve large-scale convolution calculations. This is because the planes containing matrices A, B, and the detector module have object-image conjugate relationships with each other. When a diffractive optical element (DOE) such as a Dammann grating is placed in front of the plane of matrix A, the object-image relationship of each plane can still be satisfied, and each diffraction order involved in the DOE convolution is clearly imaged. Therefore, the physical size of the matrix elements can be greatly reduced. Even commercial optical imaging systems can easily achieve high-quality imaging of matrix elements with sizes on the order of tens of micrometers, which makes this architecture have great potential to realize a highly integrated miniature optical convolution computer with ultra-high computing power and ultra-low energy consumption, comparable to a handheld camera. In addition, the higher orders of the Dammann grating can be completely filtered by the aperture without affecting the calculation results. Furthermore, by optimizing and combining Dammann gratings with different splitting ratios and high diffraction efficiencies, efficient spectral splitting of large-scale matrices with almost no energy loss can be achieved.

[0141] Compared with the various technical solutions proposed so far, the optical convolution computing system based on a multi-imaging projection architecture provided by the above embodiments of the present invention makes full use of optical two-dimensional parallelism and effectively solves the problems of low accuracy and small scale in the traditional shadow delivery method architecture. It has the advantages of good tolerance performance, high accuracy, and large scale. In future big data analysis and advanced artificial intelligence application scenarios, especially in the field of image processing with convolutional neural network technology as the core, it has significant application value and good economic benefits.

[0142] The technical solutions provided by the above embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0143] The optical convolution computation system based on the multi-imaging projection architecture proposed in this invention realizes digital optical convolution operation based on the imaging projection method (OMica). Figure 1This is a schematic diagram of an optical convolution calculation system based on the shadow method. The system includes a light source array module 101, an imaging projection module 102, Fourier transform lenses L1103a and L2103b, a focal plane 104 of the Fourier transform lens L1, a signal modulation module 105, a detection module 106, a focusing lens L3106a, and a light receiver array 106b. Specifically, the light source array module 101 loads information from the convolution kernel matrix B; the signal modulation module 105 loads information from the input matrix A; the detection module 106 detects the convolution result information of matrices A and B; the imaging projection module 102 performs copying, translation, and multiplication operations on the input matrix A using the optical signal carrying the information from the convolution kernel matrix B; and the focusing lens L3106a performs convergence and summation operations on the optical signal carrying the multiplication information of matrices A and B. Furthermore, Fourier transform lenses L1103a and L2103b form a 4F optical system. The plane containing the convolution kernel matrix B and the plane containing the input matrix A satisfy the object-image conjugate relationship. After adding the imaging projection module 102, the characteristic distance d between the imaging projection module 102 and the light source array module 101 and the diffraction order of the imaging projection module 102 are perfectly matched. Each diffraction order of the imaging projection module 102 carries all the information of the convolution kernel matrix B. Since the planes containing matrices A and B satisfy the object-image conjugate relationship, each diffraction order of the imaging projection module 102 is imaged at the corresponding position on the plane containing the input matrix A, thereby avoiding the diffraction effect introduced by the propagation distance d0 in the traditional shadow delivery method. Therefore, the optical convolution calculation system based on the multi-imaging projection architecture proposed in this invention, after loading the corresponding matrices into the light source array module 101 and the signal modulation module 105, can directly obtain the convolution result of matrices A and B on the detection module 106 after light passes through the system once. The optical convolution computing system proposed in this invention, based on a multi-imaging projection architecture, has good scalability and the potential to realize a novel optical computing system with ultra-high computing power and ultra-low power consumption.

[0144] The schematic diagram of the optical convolution computation system based on a multi-imaging projection architecture provided in the above embodiments of the present invention is as follows: Figure 2As shown, it includes an optical convolution section 21 and an electronic control section 22. The optical convolution section 21 mainly includes a light source array module 2101, an imaging projection module 2102, a signal modulation module 2103, and a detection module 2104. The light source array module 2101 is used to load the information of the convolution kernel matrix B. The light source array module can be an array of light-emitting elements, including but not limited to LEDs, LDs, fiber arrays, VCSELs, or spatial light modulators (SLMs), including but not limited to liquid crystal spatial light modulators, digital micromirror arrays (DMDs), microelectromechanical systems (MEMS), fiber arrays, and optical waveguide arrays. The signal modulation module 2103 is used to modulate the optical signal after passing through the imaging projection module 2102. The signal modulation module 2103 loads the input matrix B. The information of array A is provided by signal modulation module 2103, which includes, but is not limited to, various spatial light modulators (SLMs), such as liquid crystal spatial light modulators, digital micromirror arrays (DMDs), microelectromechanical systems (MEMS), fiber arrays, and optical waveguide arrays. Detection module 2104 is used to realize the convergence summation and detection of the convolution results. Optical signals of different orders after passing through imaging projection module 2102 are converged using optical elements such as lens L104a. The optical signals carrying the multiplication information of corresponding elements of matrix A and matrix B are converged to different positions on the detection surface of detection module 2104 according to the diffraction order to achieve the summation of the multiplication information. Finally, the data is converted by photoelectric conversion and input into a computer for further processing.

[0145] The electronic control section 22 mainly comprises a hardware system and a software control system. The hardware system includes a computing unit 2204, a data acquisition card 2203, and a phase-locked loop synchronization device 2205. The software system includes LabVIEW drivers, C language drivers, and microcontroller control software. The hardware system can realize data loading, synchronization, acquisition, digital-to-analog / analog-to-digital conversion, and photoelectric conversion. The software system can realize the automated control of the above hardware and the real-time display and processing of experimental results. The electronic control section 22 loads the convolution kernel matrix B and the input matrix A onto the corresponding light source array module 2101 and signal modulation module 2103, and controls the feature distance between the imaging projection module 2102 and the convolution kernel matrix B to perfectly match the diffraction order of the imaging projection module 2102. When the light signal passes through the system once, the convolution result matrix C of matrices A and B is directly obtained in the detection module 2104. Then, the result is acquired and processed by the computer, and the correctly decoded convolution result is output, thus completing one optical convolution calculation.

[0146] The optical convolution calculation system based on a multi-imaging projection architecture provided in the above embodiments of the present invention is described in the following description of the specific optical implementation method of the convolution process:

[0147] The convolution kernel matrix B and the input matrix A are loaded onto the light source array module 2101 and the signal modulation module 2103, respectively. The input light signal from the light source array module 2101 is loaded with the information of the convolution kernel matrix B, and then the light signal loaded with the information of the convolution kernel matrix B is copied and translated by the imaging projection module 2102. The light signal transmitted through the imaging projection module 2102 is then multiplied by the corresponding elements of the input matrix A in the signal modulation module 2103. The light source array module 2101 and the imaging projection module 2102 are distributed on two parallel planes separated by a distance d, and it is necessary to ensure that the corresponding elements of the input matrix A and the convolution kernel matrix B are spatially aligned. To further eliminate the error introduced by the zeroth order diffraction order of the imaging projection module 2102, under the condition of satisfying a specific axial distance, the light source array module 2101 and the signal modulation module 2103 are misaligned in the lateral position, so that the B matrix information loaded on the light source array module 2101 and the A matrix information on the signal modulation module 2103 are misaligned and aligned. The zero-order corresponding signal is multiplied with the zero-order signal in the signal modulation module, thus significantly eliminating the interference caused by the zero-order diffraction order of the imaging projection module 2102. The optical signal carrying the translation and replication information of the convolution kernel matrix B completes the dot product operation between the convolution kernel matrix B and the input matrix A after the input matrix A is loaded in the signal modulation module 2103.

[0148] After the dot product operation, the optical signal is converged by the lens 106a in the detection module 106 to achieve the summation operation of the dot product optical signal within the corresponding angle. The optical receiver array 106b on the focal plane behind the lens obtains the convolution result C of the convolution kernel matrix B and the input matrix A. The above two steps realize the optical convolution operation.

[0149] The elements of the spatially overlapping part of the convolution kernel matrix B and the input matrix A constitute the effective convolution part, which can participate in optical convolution operations.

[0150] A preferred specific application illustration of the optical convolution calculation system based on a multi-imaging projection architecture provided in the above embodiments of the present invention is shown below. Figure 3 As shown, the optical components are in the following order: light source array 301, Damman grating 302, first Fourier transform lens 303, second Fourier transform lens 304, spatial light modulator 305, focusing lens 306, and detector 307.

[0151] The light source array 301 is an LD light source array with a wavelength of 450nm. Its beam arrangement ratio is 526×526, and the spacing between the beams is 40μm. The light source array 301 modulates the light source array by controlling the switch of the input current and loads the information of the convolution kernel matrix B.

[0152] The Damman grating 302 has a beam splitting ratio of 1:10, a period of 1 mm, and an adjacent order diffraction angle of 0.45 mrad.

[0153] The Dammann grating 302 is placed on the back focal plane of the first Fourier transform lens 303, uniformly splitting the optical signal loaded with convolution kernel matrix B information through the light source array 301 into a 10×10 array of parallel beams. These beams diffracting in different directions form a translational sliding of the convolution kernel matrix B relative to the input matrix A. Then, the multiplication operation of the input matrix A and the convolution kernel matrix B is performed on the back focal plane of the second Fourier lens 304, that is, the plane where the light source array module is located and the plane where the input matrix A is located are conjugate planes.

[0154] The back focal plane of the first Fourier lens 303 is the front focal plane of the second Fourier lens 304, and the back focal plane of the second Fourier lens 304 is the front focal plane of the focusing lens 306. The first Fourier lens 303 has a focal length of 300mm and an aperture of 50mm, the second Fourier lens 304 has a focal length of 400mm and an aperture of 50mm, and the focusing lens 306 has a focal length of 300mm and an aperture of 50mm.

[0155] The spatial light modulator 305 has a resolution of 1920×1080 and a single pixel size of 8μm;

[0156] A single coded element of the spatial light modulator 305 is represented by a 54×54 pixel grayscale patch, which can make full use of the pixels of the spatial light modulator, and the size of each matrix unit is 432μm.

[0157] The spatial light modulator 305 loads the input matrix A by adjusting the transmittance at different locations on the control panel;

[0158] The light source array 301 loads the information of the convolution kernel matrix B. The light signal carrying the information of the convolution kernel matrix B is divided into a 10×10 array light signal by the Damman grating 302. The array light signal is incident on the end face of the spatial light modulator 305 through the 4F system to complete the dot product operation of the corresponding elements of the A and B matrices. Then, the light signal carrying the dot product information of A and B is focused on different positions 307 of the detector end face by the focusing lens 306.

[0159] The detection module consists of a detector 307 and a focusing lens 306. The detectors include, but are not limited to, CCD, CMOS, fiber-coupled detector arrays, and photodetectors.

[0160] The focal length of the focusing lens 306 is 300mm. The focusing lens 306 focuses the signal light that has been modulated by the spatial light modulator 305 and completed the multiplication operation onto different positions on the end face of the detector 307 according to different tilt angle components, so as to realize the addition operation in the convolution operation.

[0161] The detector 307 has a resolution of 1920×1200, with a single pixel size of 4.8μm. The CMOS detector receives the final system's convolution matrix C, which is the completed... Convolution operation.

[0162] Based on the working principle of the optical convolution system based on the multi-imaging projection architecture, taking the encoding of hexadecimal 4-bit matrix element values ​​into binary 1-bit matrix element values ​​as an example, the calculation process of the optical convolution calculation system based on the multi-imaging projection architecture is described according to the spatial sequence encoding:

[0163] First, the high-bit convolutional kernel matrix B202 and the input matrix A201 are represented as follows:

[0164]

[0165] According to the encoding rules of digital optical convolution systems, the low-bit matrix is ​​rearranged as follows, with pixels arranged horizontally and bits arranged vertically.

[0166]

[0167] According to the principle of spatial sequence encoding, the above low-bit matrix is ​​split and reassembled according to the bit positions. The matrix corresponding to different bit positions is as follows:

[0168]

[0169] Therefore, a convolution kernel matrix LowB is loaded onto the light source array (a). 21 bit The input matrix LowA is loaded onto the spatial light modulator 408. 11 bit Perform the first batch of operations:

[0170]

[0171] Load the LowB convolution kernel matrix onto the light source array (a). 21 bit The input matrix LowA is loaded onto the spatial light modulator 408. 12 bit Perform the second batch of calculations:

[0172]

[0173] Load the LowB convolution kernel matrix onto the light source array (a). 22 bit The input matrix LowA is loaded onto the spatial light modulator 408. 11 bit Perform the third batch of calculations:

[0174]

[0175] Load the LowB convolution kernel matrix onto the light source array (a). 22 bit The input matrix LowA is loaded onto the spatial light modulator 408. 12 bit Perform the fourth batch of calculations:

[0176]

[0177] Following the translation and sliding rules of convolution operations, the above convolution results are recombined to obtain the complete low-bit convolution result:

[0178]

[0179] The decoding method involves multiplying the element corresponding to the pixel position of the lower bit element by the corresponding base bit by bit, and then summing the results to decode the higher bit element. The decoding process is as follows:

[0180] First, rearrange the low-bit convolution results as follows:

[0181]

[0182] Then, multiply all elements at each pixel by the base corresponding to its bit position and decode to the high bit, as follows:

[0183] 6×4 2 =96.7×4 1 =28,2×4 0 =2

[0184] The high-bit result is obtained by summing the product terms at each pixel level, as follows:

[0185] 96 + 28 + 2 = 130

[0186] Similarly, by decoding the entire matrix, we can obtain:

[0187]

[0188] Based on the above encoding method, two 10×10 1-bit matrices are randomly generated and loaded onto the light source array module 2101 and the signal modulation module 2103, respectively. Figure 4 These are the results of a convolution experiment using two 10×10 1-bit randomly generated matrices. The convolution results detected on the sCMOS1 detector are as follows. Figure 4 The distribution trend shown in (a) is the same as the theoretical convolution result. The convolution result on detector sCMOS1 is converted to the corresponding grayscale value as follows: Figure 4 The theoretical value of the convolution result shown in (b) is as follows: Figure 4 As shown in (c), the error value of the convolution result is as follows: Figure 4 As shown in (d), this is the absolute error, which is the theoretical value minus the experimental value. The average error is 0.240, and all error values ​​are less than 0.5. Therefore, the correct grayscale can be obtained after performing step-level processing on the grayscale.

[0189] Figure 5 These are the results of a convolution experiment using two 10×10 1-bit matrices. The convolution result detected by the detector is shown below. Figure 5 As shown in (a), the convolution result on the detector is converted into the corresponding grayscale value, as shown in Figure (a). Figure 5 As shown in (b), the theoretical value of the convolution result is as follows: Figure 5 As shown in (c), the error between the theoretical and experimental values ​​of convolution is as follows: Figure 5 As shown in (d), the decoding result of the theoretical value of the convolution result is as follows: Figure 5 As shown in (e-1), the decoding results of the experimental values ​​of the convolution result are as follows: Figure 5 As shown in (e-2).

[0190] Figure 6 These are the experimental results of convolution based on a 10×10 1-bit matrix with digital encoding. The convolution results detected by the detector are as follows: Figure 6 As shown in (a), the convolution result on the detector is converted into the corresponding grayscale value, as shown in Figure (a). Figure 6 As shown in (b), the theoretical value of the convolution result is as follows: Figure 6 As shown in (c), the error between the theoretical and experimental values ​​of convolution is as follows: Figure 6 As shown in (d), the decoding result of the theoretical value of the convolution result is as follows: Figure 6 As shown in (e-1), the decoding results of the experimental values ​​of the convolution result are as follows: Figure 6 As shown in (e-2).

[0191] Figure 7 These are experimental results based on 20×20 1-bit matrix convolution using digital encoding. The theoretical convolution values ​​are as follows: Figure 7 As shown in (a), the light intensity distribution is as follows Figure 7 As shown in (b), the experimental convolution values ​​are as follows: Figure 7 As shown in (c), the error distributions of the two 20×20 1-bit matrices are as follows: Figure 7 As shown in (d).

[0192] Figure 8 This is a specific implementation case of an optical convolution computation system based on a multi-imaging projection architecture: experimental results of 10×10 8-bit matrix convolution based on digital encoding. The theoretical fully convolutional value is as follows: Figure 8 As shown in (a), the light intensity distribution is as follows Figure 8 As shown in (b), the experimental fully convolutional values ​​are as follows: Figure 8 As shown in (c), the error distributions of the two 10×108-bit matrices are as follows: Figure 8 As shown in (d).

[0193] Figure 9 These are experimental results based on 180×224 8-bit matrix convolution using digital encoding. The theoretical convolution results are as follows: Figure 9 As shown in (a), the experimental convolution results are as follows: Figure 9 As shown in (b), the experimental detection spot distribution is as follows: Figure 9 As shown in (c), the magnified view of the experimental probe spot shows the calculated centroid position indicated by the red cross. Figure 9 As shown in (d), the error distribution compared to the maximum value of 1 / 256 is as follows: Figure 9 As shown in (e), the circled mark indicates that the error of the corresponding convolution element value is less than 1 / 256 of the maximum value of the theoretical convolution result, and the error distribution of the convolution matrix accounts for, for example, Figure 9 As shown in (f). The above experimental results demonstrate that the optical convolution system based on the multi-imaging projection architecture proposed in this invention has good robustness and is expected to achieve larger-scale and higher-precision optical convolution calculations.

[0194] Finally, based on the above matrix convolution results, this invention is applied in a specific example of an optical convolution calculation system based on a multi-imaging projection architecture, such as... Figure 10 As shown, the optical convolution computation system based on the MNIST dataset realizes the recognition of handwritten digits. The process of performing convolution operations after encoding the convolution kernel as positive or negative is illustrated in the diagram below. Figure 10 As shown in (a), the theoretical result of the error convolution calculation between the experimental result and the error of the number 7 is as follows: Figure 10 As shown in (b), the confusion matrix of the optical convolution system based on the blind test results of 1000 images is as follows. Figure 10 As shown in (c), the computer's confusion matrix based on the blind test results of 1000 images is as follows. Figure 10As shown in (d), the 10 trained convolutional kernels are first encoded into 10 positive kernels and 1 negative kernel. The robustness and accuracy of the optical convolution system are tested using a binary neural network (BNN) as an example, where the input signal is 0 or 1 and the kernel weight is -1 or +1. After encoding the convolutional kernels, the negative kernels of the ten kernels are identical. Therefore, the total number of encoded positive and negative kernels is one more than the original number of kernels, which does not impose an additional computational burden on the optical convolution system. To demonstrate the reliability and robustness of the system, we blind-tested 1000 sets of MNIST data. The experimental results show that the recognition accuracy of the optical convolution system is as high as 97.3%, while the recognition accuracy of the electronic computer reaches 96.7%. Clearly, the error distribution is highly correlated with the input signal. Therefore, the recognition accuracy of the optical convolution system based on the MNIST dataset is slightly higher than or comparable to that of the electronic computer. However, just like the problems faced by the human eye in direct observation, the error distributions of input signals with similar distributions are also relatively close. By further upgrading the optical convolution system, the training and recognition processes of CNNs within the optical convolution system can be directly implemented, potentially achieving better results than electronic computers. Based on this, the optical convolution system can be effectively used as a powerful hardware accelerator for various optical neural networks (ONNs).

[0195] In summary, compared with traditional optical convolution schemes or existing optical computing schemes, the optical convolution computing system based on a multi-imaging projection architecture proposed in this invention achieves large-scale, high-precision, and highly parallel optical convolution calculations. By introducing an imaging projection module, it effectively utilizes the diffraction effect of the Dammann grating to achieve large-scale matrix convolution calculations. This is because the planes containing the A matrix, B matrix, and detector have a conjugate object-image relationship. When a diffractive optical element (DOE) such as a Dammann grating is placed in front of the A matrix plane, the object-image relationship of each plane is still satisfied, and each diffraction order involved in the DOE convolution is clearly imaged. Furthermore, higher orders of the Dammann grating can be completely filtered by the aperture without affecting the calculation results. By optimizing and combining Dammann gratings with different splitting ratios, efficient spectral splitting of larger matrices with almost no energy loss can be achieved. Therefore, the physical size of the matrix elements can be greatly reduced. Even in commercial optical imaging systems, high-quality imaging of matrix elements with sizes on the order of tens of micrometers can be easily achieved. Finally, based on a suitable numerical encoding algorithm, by optimizing the optical system and using higher-contrast SLMs (including MEMS and DMD), higher precision and larger-scale convolution calculations can be achieved. Assuming matrices A and B are of size N×N, the computational power can be estimated as: O = (N×N + (N×N-1)(N×N)) ≈ 2N 4If a 1K (1024×1024) spatial light modulator is used, with a refresh rate of 20kHz, the computing power is 1M TOP / s. If 100 layers of optical convolution calculation are implemented based on this system, the computing power is 100×1M TOP / s=100M TOPs.

[0196] Furthermore, if high-dimensional multiplexing methods, including those involving polarization, wavelength, and mode, are used to enhance optical communication capacity, speeds at least 10 times faster than the aforementioned estimates can be achieved. 2 Up to 10 3 The proposed optical convolutional computing architecture achieves a multiplier increase in computing power. By using high-performance devices (such as large modulators with higher update frequencies, detectors or detector arrays with wider dynamic ranges and higher sampling frequencies) to progressively improve the computational power and energy efficiency of convolution, the system can continuously improve computing power and accuracy, and has good scalability, opening a new door to realizing large-scale, high-precision optical convolutional computing.

[0197] In some embodiments of the present invention:

[0198] This optical convolution computation system based on a multi-imaging projection architecture can achieve... High-precision, large-scale convolution computation, where A is the input matrix, B is the convolution kernel matrix, and C is the convolution result matrix. This represents a convolution operation, which involves multiplying and summing the elements of the convolution kernel matrix B with the corresponding elements of the input matrix A. The convolution kernel slides across the other matrix with a fixed stride, repeatedly performing the multiplication and summation operations until all positions are traversed to obtain the convolutional matrix. The output matrix C can be a fully convolutional matrix or a matrix of the same size as the input matrix A.

[0199] The light source array module is placed before the imaging projection module, and the light source array module is used to load the information of the convolution kernel matrix B.

[0200] The imaging projection module is positioned after the light source array module and before the signal modulation module. The imaging projection module is used to translate and replicate the optical signal carrying the information of the convolution kernel matrix B on the signal modulation module. Specifically, the optical signals of different diffraction orders passing through the imaging projection module are projected at different angles θ. p The optical signals of different diffraction orders are transmitted to different positions on the plane of the signal modulation module. Here, p is the index of the diffraction order. Each diffraction order optical signal carries all the information of the convolution kernel matrix B and is multiplied with the information of the corresponding position of the matrix A loaded on the signal modulation module. The optical signals of different diffraction orders of the imaging projection module are propagated to different positions on the plane of the signal modulation module, which corresponds to the translation, sliding and multiplication operations of the convolution kernel matrix B on the input matrix A in the convolution calculation.

[0201] The signal modulation module is positioned after the imaging projection module and before the detection module. It modulates optical signals carrying information about the convolution kernel matrix B after passing through the imaging projection module, based on different diffraction orders. The signal modulation module also loads information about the input matrix A. The detection module is positioned after the signal modulation module. It performs convergence summation and detection of the convolution results. Through optical elements such as lenses in the detection module, the optical signals containing the element-wise multiplication information of matrices B and A at different positions are converged to different positions on the detection surface of the detection module according to the tilt components of the diffraction orders, achieving the summation operation. The lens elements in the detection module converge the optical signals carrying the multiplication information of A and B, corresponding to the element-wise multiplication operation of the convolution kernel matrix B and the input matrix A in the convolution calculation.

[0202] This optical convolution computing system based on a multi-imaging projection architecture can perform both analog and digital convolution operations. When performing analog convolution calculations, the input matrix A and convolution kernel matrix B are analog quantities, and the convolution result matrix C obtained from the detection module is also an analog quantity. After digital processing, the convolution result matrix C is quantized into a digital result. When performing high-precision, large-scale digital optical convolution calculations, the system uses matrix encoding and decoding algorithms to represent the high-bit matrix to be processed as multiple encoded low-bit matrices. The encoded low-bit convolution kernel matrix B and the low-bit input matrix A are then loaded onto the light source array module and signal modulation module, respectively. When light passes through the system once, the low-bit convolution result matrix C is obtained on the detection surface of the detection module. Finally, the low-bit convolution result matrix is ​​decoded into a high-bit matrix.

[0203] The electronic control components section includes hardware components and control components. The hardware components include a precision displacement platform, a servo drive motor, a data acquisition card, and a phase-locked loop synchronization device. The control components include a LabVIEW driver module, a C language driver module, and a microcontroller control module. The hardware components can realize data loading, synchronization, acquisition, digital-to-analog / analog-to-digital conversion, and photoelectric conversion. The control components can realize the automated control of the above hardware components and the real-time display and processing of experimental results.

[0204] The light source array module can be an array of light-emitting elements, including but not limited to LEDs, LDs, fiber arrays, and vertical cavity surface-mount semiconductor laser arrays (VCSELs), or it can be a spatial light modulator (SLM), including but not limited to liquid crystal spatial light modulators (LCSLMs), digital micromirror arrays (DMDs), microelectromechanical systems (MEMS), fiber arrays, and optical waveguide arrays.

[0205] The imaging projection module can be a one-dimensional Damman grating with a beam splitting ratio of 1×3 to 1×128, or a two-dimensional Damman grating with a beam splitting ratio of 3×3 to 128×128 or larger. By combining Damman gratings, higher diffraction efficiency and larger-scale beam splitting effect can be achieved. It can also be other beam splitting elements, including but not limited to diffractive optical elements (DOE), metasurfaces and metamaterials.

[0206] Signal modulation modules include, but are not limited to, various spatial light modulators (SLMs), such as liquid crystal spatial light modulators (LDSLMs), digital micromirror arrays (DMDs), microelectromechanical systems (MEMS), fiber optic arrays, and optical waveguide arrays.

[0207] The detection module includes at least one lens and an optical receiver array. The lens is used to converge and sum optical signals of different diffraction orders at different positions, and the optical receiver array is used to detect optical convolution signals, including but not limited to CMOS and CCD detectors.

[0208] The optical convolution computing system based on a multi-imaging projection architecture provided in the above embodiments of the present invention can be optimized and developed based on any form of optical system that satisfies the object-image conjugate relationship between the convolution kernel matrix B and the input matrix A. It can be a transmissive optical system, a refractive optical system, or a reflective optical system. It can be a paraxial optical system, an off-axis optical system, or a planar space optical waveguide system. It includes, but is not limited to, various variations and combinations of single or multiple cascaded 4F canonical signal systems, microscopic systems, telescope systems, projection systems, and the above systems.

[0209] The optical convolution computing system based on a multi-imaging projection architecture provided in the above embodiments of the present invention can further improve computing power through novel optical communication technologies that expand capacity, including but not limited to wavelength division multiplexing, mode division multiplexing, and partial division multiplexing.

[0210] Figure 11 This is a flowchart of an optical convolution calculation method based on a multi-imaging projection architecture, provided as an embodiment of the present invention.

[0211] like Figure 11 As shown, the optical convolution calculation method based on a multi-imaging projection architecture provided in this embodiment may include the following steps:

[0212] S100: Load the convolution kernel matrix information onto the input optical signal to obtain an optical signal carrying the convolution kernel matrix information;

[0213] S200 generates optical signals of different diffraction orders from optical signals carrying convolution kernel matrix information;

[0214] S300: Load the input matrix information, and multiply the convolution kernel matrix information carried by the optical signals of different diffraction orders with the input matrix information to obtain the multiplication information of the corresponding elements of the convolution kernel matrix information and the input matrix information at different positions, thereby obtaining the optical signal carrying the multiplication information.

[0215] S400 converges the optical signals carrying multiplication information, performs a summation operation on the multiplication information, and obtains the convolution result matrix information.

[0216] In a preferred embodiment, in S200, optical signals of different diffraction orders are transmitted to different positions of the input matrix at different angles, wherein each optical signal of a diffraction order carries all the information of the convolution kernel matrix.

[0217] In a preferred embodiment, step S300 involves multiplying the convolution kernel matrix information with the input matrix information, including:

[0218] S301, multiply the elements in the convolution kernel matrix information with the corresponding elements in the input matrix information;

[0219] S302, slide the elements in the convolution kernel matrix information into the input matrix information with a fixed stride and repeat the above multiplication steps;

[0220] S303, after traversing all positions, obtains the multiplication information of the corresponding elements of the convolution kernel matrix and the input matrix at different positions.

[0221] In a preferred embodiment, S400 includes a summation operation on the multiplied information, including:

[0222] The optical signals carrying multiplication information are focused onto different positions on the detection surface of the detection module according to the tilt components of the diffraction order, so as to realize the summation operation of the multiplication information.

[0223] In a preferred embodiment, multiplying the elements in the convolution kernel matrix information with the corresponding elements in the input matrix information includes:

[0224] Multiply the elements in the convolution kernel matrix with the corresponding elements in the input matrix:

[0225] {[a 11 *b jk a 12 *b j(k+1) , ..., a 1n *b j(k+n-1) ];

[0226] [a 21 *b (j+1)k ,a22 *b (j+1)(k+1) , ..., a 2n *b (j+1)(k+n-1) ];

[0227] …

[0228] [a m1 *b (j+m-1)k a m2 *b (j+m-1)(k+1) , ..., a mn *b (j+m-1)(k+n-1) ]};

[0229] Among them, a 11 a 12 , ..., a mn Let b be an element of the input matrix. 11 b 12 , ..., b mn The elements are the convolution kernel matrix, with subscripts m and n being the row and column numbers of the kernel matrix, respectively, and subscripts j and k being the row and column numbers of the overlapping elements of the input matrix and the kernel matrix, respectively.

[0230] In a preferred embodiment, the elements in the convolution kernel matrix information of the multiplication information are summed with the corresponding elements in the input matrix information:

[0231] {c 11 =[a 11 *b 11 +a 12 *b 12 +…+a 1n *b 1n ]+[a 21 *b 21 +a 22 *b 22 +…+a 2n *b 2n ]+…+[a m1 *b m1 +a m2 *b m2 +…+a mn *b mn ];

[0232] c 1s =[a 1(1+s) *b 1(1+s) +a 1(2+s) *b 1(2+s) +…+a 1(n+s) *b 1(n+s) ]+[a 2(1+s) *b 2(1+s) +a 2(2+s) *b 2(2+s)+…+a 2(n+s) *b 2(n+s) ]+…+[a m(1+s) *b m(1+s) +a m(2+s) *b m(2+s) +…+a m(n+s) *b m(n+s) ];

[0233] …

[0234] c ts =[a (1+t)(1+s) *b (1+t)(1+s) +a (1+t)(2+s) *b (1+t)(2+s) +…+a (1+t)(n+s) *b (1+t)(n+s) ]+[a (2+t)(1+s) *b (2+t)(1+s) +a (2+t)(2+s) *b (2+t)(2+s) +…+a (2+t)(n+s) *b (2+t)(n+s) ]+…+[a (m+t)(1+s) *b (m+t)(1+s) +a (m+t)(2+s) *b (m+t)(2+s) +…+a (m+t)(n+s) *b (m+t)(n+s) ]};

[0235] Where the subscripts t and s represent the lengths of the horizontal and vertical sliding steps of the convolution kernel matrix on the input matrix, respectively.

[0236] After summing, we obtain the convolution result matrix information of the convolution kernel matrix information and the input matrix information.

[0237] In a preferred embodiment, the loaded convolution kernel matrix information, the loaded input matrix information, and the obtained convolution result matrix information are all analog quantities. The convolution result matrix information is quantized into a digital result after digital processing to realize analog convolution operation.

[0238] In a preferred embodiment, the high-bit convolution kernel matrix information and the high-bit input matrix information to be processed are represented as multiple encoded low-bit matrices, respectively, to obtain encoded low-bit convolution kernel matrix information and low-bit input matrix information; the low-bit convolution kernel matrix information and the low-bit input matrix information are respectively used as loaded matrix information to obtain low-bit convolution result matrix information; the low-bit convolution result matrix information is decoded into high-bit convolution result matrix information to realize digital convolution operation.

[0239] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the system to implement the steps and flow of the method. That is, the embodiments in the system can be understood as preferred examples of the method, and will not be elaborated here.

[0240] The optical convolution calculation system and method based on a multi-imaging projection architecture provided in the above embodiments of the present invention achieve high parallelism and high precision convolution calculation of the convolution kernel matrix B and the input matrix A. Using the technical solution provided in the above embodiments of the present invention, by simply loading the convolution kernel matrix B and the input matrix A onto the corresponding light source array module and signal modulation module, a large-scale, high-precision convolution operation result matrix C can be directly obtained on the detection module after the optical signal passes through the optical convolution calculation system in a single pass. The optical convolution calculation system and method based on a multi-imaging projection architecture proposed in the above embodiments of the present invention can realize a new technical route for large-scale, high-precision, and fully parallel optical computing, providing a general and efficient solution to meet the needs of artificial intelligence, neural network image processing, and other tasks for massive convolution operations.

[0241] The optical convolution calculation system and method based on a multi-imaging projection architecture disclosed above represent only one specific embodiment of the present invention and should not be construed as limiting the scope of protection of the present invention. It should be noted that those skilled in the art can make several non-inventive modifications and improvements to the specific implementation details and representative devices proposed in this patent without departing from the basic idea of ​​the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

[0242] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples, without contradiction. Having described the invention in such detail and with reference to its preferred embodiments, it will be apparent that modifications and variations are possible without departing from the scope of the invention as defined in the appended claims.

[0243] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. An optical convolution computing system based on a multi-imaging projection architecture, characterized in that, The application relates to an optical convolution component, which comprises: an optical source array module, an imaging projection module arranged at the rear end of the optical source array module, a signal modulation module arranged at the rear end of the imaging projection module, and a detection module arranged at the rear end of the signal modulation module; wherein: the optical source array module is used for loading convolution kernel matrix information to input optical signals to obtain optical signals carrying the convolution kernel matrix information; the imaging projection module is used for generating optical signals of different diffraction orders from the optical signals carrying the convolution kernel matrix information and outputting the optical signals to the signal modulation module; the signal modulation module is used for loading input matrix information and performing multiplication operation on the convolution kernel matrix information carried by the optical signals of different diffraction orders and the input matrix information to obtain multiplication information of corresponding elements of the convolution kernel matrix information and the input matrix information at different positions, and then obtain optical signals carrying the multiplication information; and the detection module is used for converging the optical signals carrying the multiplication information to realize summation operation on the multiplication information and obtain convolution result matrix information. The imaging projection module comprises a plurality of beam splitting devices, and the plurality of beam splitting devices divide the optical signals carrying the convolution kernel matrix information into a plurality of sub-beams; wherein each sub-beam is incident to different positions of the signal modulation module according to respective diffraction angles to realize parallel translation operation of the optical signals carrying the convolution kernel matrix information; and the convolution kernel matrix information carried in each sub-beam incident to the signal modulation module is respectively multiplied with the input matrix information at the pixel level to realize parallel multiplication operation of the convolution kernel matrix information and the input matrix information. The parallel multiplication operation of the convolution kernel matrix information and the input matrix information comprises: multiplying elements in the convolution kernel matrix information with elements at corresponding positions in the input matrix information; repeatedly executing the above multiplication step after the elements in the convolution kernel matrix information slide in the input matrix information according to a fixed step; and obtaining multiplication information of corresponding elements of the convolution kernel matrix information and the input matrix information at different positions after traversing all positions. The summation operation on the multiplication information comprises: converging the optical signals carrying the multiplication information to different positions of a detection surface of the detection module according to diffraction order inclination components to realize summation operation on the multiplication information. The optical source array module simultaneously loads one or more convolution kernel matrix information by using one or more optical signals with different characteristics, the optical signals carrying one or more convolution kernel matrix information with different characteristics are respectively multiplied with input matrix information after passing through the imaging projection module and the signal modulation module, and finally converge through the detection module to obtain convolution result matrix information of the one or more convolution kernel matrix information and the input matrix information. The plane where the optical source array module is located and the plane where the signal modulation module is located satisfy object-image conjugate relationship. ​ ​ ​ ​ ​ ​ ​ 2. The optical convolutional computing system based on a multi-imaging projection architecture of claim 1, wherein, ​ 3. The optical convolutional computing system based on a multi-imaging projection architecture of claim 1, wherein, ​ 4. The optical convolutional computing system based on a multi-imaging projection architecture of claim 1, wherein, By adjusting the feature distance between the imaging projection module and the light source array module, the scattering angle of each diffraction order of the light source array module and the imaging projection module is matched, thereby changing the sliding step of the convolution kernel matrix information on the input matrix information.

5. The optical convolutional computing system based on a multi-imaging projection architecture of claim 1, wherein, The light source array module and the signal modulation module are misaligned in the lateral position, so that the convolution kernel matrix information and the input matrix information are misaligned.

6. The optical convolutional computing system based on a multi-imaging projection architecture of claim 1, wherein, The system further comprises any one or any combination of the following: The light source array module uses an array of light emitting elements or a spatial light modulator; The imaging projection module comprises one or more Dammann gratings, which use one-dimensional or two-dimensional Dammann gratings; The imaging projection module uses a metasurface light splitting element or a metamaterial light splitting element; The signal modulation module uses a spatial light modulator; The detection module comprises a lens and an array of light receivers, wherein the lens is used to converge optical signals of different diffraction orders to different positions according to a specific diffraction angle to perform summation operation, and the array of light receivers is used to realize detection of the convolution result matrix information.

7. The optical convolutional computing system based on multi-imaging projection architecture of any of claims 1-6, wherein, Further comprising: An electronic control assembly; The electronic control assembly comprises a hardware assembly and a control assembly; wherein: The hardware assembly comprises a precision displacement platform, a servo drive motor, a data acquisition card, and a phase-locked synchronization device, wherein the servo drive motor is used to control the moving step of the precision displacement platform, the precision displacement platform is used to control the multi-dimensional precision adjustment of all functional modules in the optical convolution assembly, the data acquisition card is used to acquire the convolution result matrix information detected by the detection module, and the phase-locked synchronization device is used to realize the synchronous uploading and downloading of loaded information in the light source array module, the signal modulation module, and the detection module; The control assembly comprises a Labview driving module, a C language driving module, and a single-chip microcomputer control module, wherein the Labview driving module is used to provide a graphical human-computer interaction interface, the C language driving module is used to define operation instructions of the hardware assembly, and the single-chip microcomputer control module controls the hardware assembly according to the operation instructions defined by the C language driving module.

8. An optical convolution computation method based on a multi-imaging projection architecture, characterized in that, Including: Loading convolution kernel matrix information on input optical signals to obtain optical signals carrying convolution kernel matrix information; Generating optical signals of different diffraction orders from the optical signals carrying convolution kernel matrix information; Loading input matrix information and performing multiplication operation on the convolution kernel matrix information carried by the optical signals of different diffraction orders and the input matrix information to obtain multiplication information of corresponding elements of the convolution kernel matrix information and the input matrix information at different positions, and then obtain optical signals carrying the multiplication information; Converging the optical signals carrying the multiplication information to realize summation operation on the multiplication information and obtain convolution result matrix information; The optical signals of different diffraction orders are transmitted to different positions of the input matrix at different angles, wherein each diffraction order of optical signals carries all information of the convolution kernel matrix. The multiplication operation of the convolution kernel matrix information and the input matrix information comprises: multiplying an element in the convolution kernel matrix information with an element at a corresponding position in the input matrix information; repeating the multiplication step by sliding the element in the convolution kernel matrix information in the input matrix information according to a fixed step length; after traversing all positions, obtaining the multiplication information of corresponding elements in the convolution kernel matrix information and the input matrix information at different positions; the summation operation of the multiplication information comprises: converging optical signals carrying the multiplication information to different positions on a detection surface of a detection module according to the inclination components of diffraction orders, so as to realize the summation operation of the multiplication information.

9. The optical convolution computation method based on multi-imaging projection architecture according to claim 8, wherein, The multiplication operation of the convolution kernel matrix information and the input matrix information comprises: multiplying an element in the convolution kernel matrix information with an element at a corresponding position in the input matrix information; {[a 11 *b jk , a 12 *b j(k+1) ,..., a 1n *b j(k+n-1) ] [a 21 *b (j+1)k ,a 22 *b (j+1)(k+1) , …, a 2n *b (j+1)(k+n-1) ] … [a m1 *b (j+m-1)k , a m2 *b (j+m-1)(k+1) ,..., a mn *b (j+m-1)(k+n-1) ]}; wherein a 11 , a 12 , …, a mn are elements of the input matrix, b 11 , b 12 , …, b mn are elements of the convolution kernel matrix, and the subscripts m, n are the number of rows and columns of the convolution kernel matrix, respectively, and the subscripts j, k are the number of rows and columns of the overlapping portion of the input matrix and the convolution kernel matrix, respectively. the summation operation of the multiplication information of the element in the convolution kernel matrix information and the element at the corresponding position in the input matrix information; {c 11 = [a 11 *b 11 + a 12 *b 12 +... + a 1n *b 1n ] + [a 21 *b 21 + a 22 *b 22 +... + a 2n *b 2n ] +... + [a m1 *b m1 + a m2 *b m2 +... + a mn *b mn ] c 1s = [a 1(1+s) *b 1(1+s) +a 1(2+s) *b 1(2+s) +…+a 1(n+s) *b 1(n+s) ]+[a 2(1+s) *b 2(1+s) +a 2(2+s) *b 2(2+s) +…+a 2(n+s) *b 2(n+s) ]+…+[a m(1+s) *b m(1+s) +a m(2+s) *b m(2+s) +…+a m(n+s) *b m(n+s) ] … c ts = [a (1+t)(1+s) *b (1+t)(1+s) + a (1+t)(2+s) *b (1+t)(2+s) +... + a (1+t)(n+s) *b (1+t)(n+s) ] + [a (2+t)(1+s) *b (2+t)(1+s) + a (2+t)(2+s) *b (2+t)(2+s) +... + a (2+t)(n+s) *b (2+t)(n+s) ] +... + [a (m+t)(1+s) *b (m+t)(1+s) + a (m+t)(2+s) *b (m+t)(2+s) +... + a (m+t)(n+s) *b (m+t)(n+s) ]}; wherein, the subscripts t and s are the lengths of the horizontal and vertical sliding step lengths of the convolution kernel matrix on the input matrix.

10. The optical convolution calculation method based on multi-imaging projection architecture according to any one of claims 8-9, characterized in that, The loaded convolution kernel matrix information, the loaded input matrix information and the obtained convolution result matrix information are all analog quantities, and the convolution result matrix information is quantized to a digital result after digital processing, so as to realize analog convolution operation.

11. The optical convolution calculation method based on multi-imaging projection architecture according to any one of claims 8-9, wherein, The high-bit convolution kernel matrix information and the high-bit input matrix information to be processed are respectively represented as a plurality of encoded low-bit matrixes, so as to obtain encoded low-bit convolution kernel matrix information and low-bit input matrix information; the low-bit convolution kernel matrix information and the low-bit input matrix information are respectively taken as loaded matrix information, so as to obtain low-bit convolution result matrix information; and the low-bit convolution result matrix information is decoded as high-bit convolution result matrix information, so as to realize digital convolution operation.

Images