High-dimensional tensor convolution accelerator based on thin-film lithium niobate

Abstract

The invention discloses a high-dimensional tensor convolution accelerator based on thin film lithium niobate. Comprising a light source array (1), a first film lithium niobate electro-optic intensity modulator array (2), a wavelength division multiplexer (3), a beam splitter (4), a second film lithium niobate electro-optic intensity modulator array (5), a wavelength division demultiplexer array (6), a balanced detector array (7) and a charge integrator array (8) which are connected in sequence. According to the method, multi-channel parallel convolution calculation is adopted, high-dimensional image or video information is processed at the same time, repeated calculation areas and memory access can be reduced, and calculation efficiency is improved; the thin-film lithium niobate high-speed electro-optical modulator is used for loading data, so that the calculation speed of the chip can be obviously improved; and the high-dimensional tensor convolution accelerator has the characteristic of high integration, so that the energy efficiency ratio of a calculation chip can be further improved.

Description

Technical Field

[0001] This invention relates to the field of optical computing technology, and in particular to a high-dimensional tensor convolution accelerator based on thin-film lithium niobate. Background Technology

[0002] As artificial intelligence (AI) achieves increasingly complex functions, it places higher demands on computing hardware in terms of computing power and energy efficiency. Traditional electronic computing technology, limited by its physical characteristics, has gradually revealed numerous bottlenecks over its long development. When computing tasks become increasingly complex and data volumes increase dramatically, the "Von Neumann bottleneck" problem in electronic computing systems becomes prominent—the transmission speed between data processing and data storage becomes a key factor restricting computing efficiency, severely impacting overall system performance. Electronic computing is approaching the limits of Moore's Law, constrained in terms of computing power, energy consumption, and latency, making it difficult to support the sustainable development of AI. Optical computing, as a highly promising emerging computing paradigm, has emerged to address this challenge. Using photons as the computing medium, photonic computing possesses advantages that can achieve a disruptive improvement in computing performance. Photons themselves travel at the speed of light, generate no heat, possess more dimensions, and can load information in parallel. Photonic computing features low power consumption, high-speed computing, and parallel computing capabilities, enabling significant improvements in computing speed and energy efficiency.

[0003] Currently, on-chip integrated optical convolutional computing hardware platforms are primarily silicon-based. The essence of convolution is sliding a convolution kernel across the input data, integrating local information from different locations into a feature map. Due to inherent redundancy and computational waste in convolutional computation, the hardware deployment efficiency of convolution faces significant challenges. Redundancy stems from the overlapping nature of convolution operations, requiring numerous multiplications and additions for the same input samples. Especially when processing images using convolution, a small convolution kernel sliding across a large dataset results in significant redundancy. This problem arises from the overlap between the convolution kernel and adjacent data segments, leading to repeated multiplications and additions in these overlapping regions. Using multiple devices or clock cycles for these calculations results in inefficient resource utilization, limiting real-time processing capabilities. Furthermore, silicon electro-optic modulators based on the plasma dispersion effect have inherent limitations, including limited response speed, nonlinear electro-optic characteristics, and significant carrier absorption losses. In optical matrix multiplication computation, computational speed is positively correlated with data transmission rate, and the modulator is the primary device affecting the data transmission rate. Summary of the Invention

[0004] Technical Problem: The purpose of this invention is to provide a high-dimensional tensor convolution accelerator based on thin-film lithium niobate, which employs multi-channel parallel convolution computation to process high-dimensional image or video data simultaneously, and has highly integrated characteristics, solving the problems of redundancy, low computation speed, and low computational energy efficiency in existing tensor convolution computations.

[0005] Technical solution: The present invention provides a high-dimensional tensor convolution accelerator based on thin-film lithium niobate, comprising a light source array, a first thin-film lithium niobate electro-optic intensity modulator array, a wavelength division multiplexer, a beam splitter, a second thin-film lithium niobate electro-optic intensity modulator array, a wavelength division multiplexer array, a balanced detector array, and a charge integrator array connected in sequence.

[0006] in,

[0007] The light source array is used to generate multiple optical carrier signals of different wavelengths;

[0008] The first thin-film lithium niobate electro-optic intensity modulator array is used to load information to be processed and to modulate the optical carrier signal according to the loaded information to be processed to obtain an optical intensity signal carrying the information to be processed.

[0009] The wavelength division multiplexer is used for multiplexing multiple single-wavelength optical carriers;

[0010] The beam splitter is used to evenly distribute the optical carrier power;

[0011] The second thin-film lithium niobate electro-optic intensity modulator array is used to load weight information, modulate the optical intensity signal output by the first thin-film lithium niobate electro-optic intensity modulator array according to the loaded weight information, and combine it with a balanced detector to encode and detect negative weights, thereby obtaining an optical carrier signal carrying the information to be processed and the weight information.

[0012] The wavelength demultiplexer array is used for multi-wavelength optical carrier demultiplexing;

[0013] The balance detector is used to detect the calculation results of positive or negative weights;

[0014] The charge integrator is used to accumulate the signals detected by the detector.

[0015] The number of light sources N in the light source array is a positive integer greater than or equal to 3; and it is integrated on a thin-film lithium niobate chip using a heterogeneous integration process.

[0016] The first thin-film lithium niobate electro-optic intensity modulator array uses a 1×2 multimode interference coupler (MMI) to split and combine the light from the two arms of the input modulator. The number of modulators in the first thin-film lithium niobate electro-optic intensity modulator array is the same as the number of light sources, and it is a push-pull Mach-Zehnder electro-optic intensity modulator.

[0017] The wavelength division multiplexer has one N×1 wavelength division multiplexer, where N is a positive integer greater than or equal to 3; the wavelength demultiplexer array has 2M 1×N wavelength demultiplexers, where M is a positive integer greater than or equal to 3.

[0018] The second thin-film lithium niobate electro-optic intensity modulator array is a push-pull Mach-Zehnder electro-optic intensity modulator containing M modulators, where M is a positive integer greater than or equal to 3; a Y-branch beam splitter is used to split the light into two arms of the input modulator; a 2×2 MMI is used to interfere the two beams of light passing through the modulator arms, and finally the light is output from the two ports of the MMI.

[0019] The balanced detector array consists of N×M units and is integrated onto a thin-film lithium niobate chip using a flip-chip bonding process, where M is a positive integer greater than or equal to 3 and N is a positive integer greater than or equal to 3.

[0020] The charge integrator array consists of N×M units, where M is a positive integer greater than or equal to 3 and N is a positive integer greater than or equal to 3. It is integrated onto a thin-film lithium niobate chip using heterogeneous integration technology.

[0021] The information to be processed includes high-dimensional tensor image or video data. Loading the information to be processed specifically includes: firstly, separating the image tensor information into R, G, and B three-channel matrix data; then flattening the separated three-channel matrix data into vectors, and loading the three-channel data onto the three modulators in the first thin-film lithium niobate electro-optic intensity modulator array respectively.

[0022] Alternatively: First, the video tensor information is separated into four-channel matrix data: R, G, B, and t. Then, the separated four-channel matrix data is flattened into vectors, and the four-channel data is loaded onto the four modulators in the first thin-film lithium niobate electro-optic intensity modulator array. Here, R, G, B, and t represent red, green, blue, and time, respectively.

[0023] In the second thin-film lithium niobate electro-optic intensity modulator array, for image convolution calculation, every three modulators form a convolution kernel, and each modulator is loaded with the weight information of a single channel; for video convolution calculation, every four modulators form a convolution kernel, and each modulator is loaded with the weight information of a single channel; different numbers of convolution kernels are set according to different processing tasks, and the number of convolution kernels is greater than or equal to 1.

[0024] The number of images or videos loaded in the first thin-film lithium niobate electro-optic intensity modulator array for image convolution calculation is greater than or equal to 1.

[0025] The loading of weight information specifically involves: flattening the weight information of a single channel in the convolution kernel into a vector, and then loading the weight data of a single channel into each modulator in the second thin-film lithium niobate electro-optic intensity modulator array.

[0026] Beneficial effects: Compared with existing technologies, the present invention has the following advantages:

[0027] 1. The high-dimensional tensor convolution accelerator based on thin-film lithium niobate proposed in this invention uses a thin-film lithium niobate high-speed electro-optic intensity modulator to load computational data. Compared with other materials or other forms of modulation, the data transmission rate is significantly improved, thereby increasing the computation speed.

[0028] 2. The high-dimensional tensor convolution accelerator based on thin-film lithium niobate proposed in this invention separates high-dimensional image or video data into different channels for calculation through multi-channel parallel convolution calculation, which greatly reduces data redundancy during the convolution operation, and the parallel calculation can further improve the calculation speed.

[0029] 3. The high-dimensional tensor convolution accelerator based on thin-film lithium niobate proposed in this invention has a computing chip based on thin-film lithium niobate, which realizes a high degree of integration of devices including light source array, modulator array, wavelength division multiplexer / demultiplexer, balanced detector, etc., and is expected to achieve optical computing with ultra-low latency and ultra-high energy efficiency. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of a high-dimensional tensor convolution accelerator based on thin-film lithium niobate according to the present invention.

[0031] Figure 2 This is a schematic diagram of a high-dimensional tensor convolution accelerator testing system based on thin-film lithium niobate according to the present invention.

[0032] Figure 3 This is a schematic diagram of the calculation process of a high-dimensional tensor convolution accelerator based on thin-film lithium niobate according to the present invention.

[0033] The diagram includes: light source array 1, first thin-film lithium niobate electro-optic intensity modulator array 2, wavelength division multiplexer 3, beam splitter 4, second thin-film lithium niobate electro-optic intensity modulator array 5, wavelength division multiplexer array 6, balanced detector array 7, charge integrator array 8, electrical control module 9, and computer 10. Detailed Implementation

[0034] To better understand the purpose, structure, and function of this invention, the following detailed description of a high-dimensional tensor convolution accelerator based on thin-film lithium niobate is provided in conjunction with the accompanying drawings.

[0035] Please see Figure 1 The present invention provides a high-dimensional tensor convolution accelerator based on thin-film lithium niobate, comprising: a light source array 1, a first thin-film lithium niobate electro-optic intensity modulator array 2, a wavelength division multiplexer 3, a beam splitter 4, a second thin-film lithium niobate electro-optic intensity modulator array 5, a wavelength division multiplexer array 6, a balanced detector array 7, and a charge integrator array 8 connected in sequence.

[0036] in,

[0037] Light source array 1 is used to generate multiple optical carrier signals of different wavelengths;

[0038] The first thin-film lithium niobate electro-optic intensity modulator array 2 is used to load the information to be processed and modulate the optical carrier signal according to the loaded information to be processed to obtain an optical intensity signal carrying the information to be processed.

[0039] Wavelength division multiplexer 3 is used for multiplexing multiple single-wavelength optical carriers;

[0040] Beam splitter 4 is used to evenly distribute the optical carrier power;

[0041] The second thin-film lithium niobate electro-optic intensity modulator array 5 is used to load weight information, modulate the optical intensity signal output by the first thin-film lithium niobate electro-optic intensity modulator array 2 according to the loaded weight information, and combine it with the balanced detector array to encode and detect negative weights, thereby obtaining an optical carrier signal carrying the information to be processed and the weight information.

[0042] Wavelength demultiplexer array 6, used for multi-wavelength optical carrier demultiplexing;

[0043] The balanced detector array 7 is used to detect the calculation results of positive or negative weights;

[0044] The charge integrator array 8 is used to accumulate the signals detected by the detector.

[0045] Optionally, the number N of light sources in the light source array 1 is a positive integer greater than or equal to 3; and it is integrated on a thin-film lithium niobate chip using a heterogeneous integration process.

[0046] Optionally, the number of modulators in the first thin-film lithium niobate electro-optic intensity modulator array 2 is the same as the number of light sources.

[0047] Optionally, the first thin-film lithium niobate electro-optic intensity modulator array 2 and the second thin-film lithium niobate electro-optic intensity modulator array 5 are push-pull Mach-Zehnder electro-optic intensity modulators.

[0048] Optionally, in the first thin-film lithium niobate electro-optic intensity modulator array 2, a 1×2 MMI is used to split and combine the light from the two arms of the input modulator.

[0049] Optionally, the second thin-film lithium niobate electro-optic intensity modulator array 5 includes M modulators, where M is a positive integer greater than or equal to 3.

[0050] Optionally, in the second thin-film lithium niobate electro-optic intensity modulator array 5, a Y-branch beam splitter is used to split the light into two arms of the input modulator; a 2×2 MMI is used to interfere the two beams of light passing through the modulation arms, and finally the light is output from the two ports of the MMI.

[0051] Optionally, wavelength division multiplexer 3 has one N×1 wavelength division multiplexer; wavelength demultiplexer array 6 has 2M 1×N wavelength demultiplexers.

[0052] Optionally, the balanced detector array 7 consists of N×M arrays and is integrated onto a thin-film lithium niobate chip using a flip-chip bonding process;

[0053] Optionally, the charge integrator array 8 comprises N×M units and is integrated onto a thin-film lithium niobate chip using heterogeneous integration technology.

[0054] It should be noted that all thin-film lithium niobate electro-optic intensity modulators must operate at the linear bias point.

[0055] Please see Figure 2 The present invention discloses a test system for a high-dimensional tensor convolution accelerator based on thin-film lithium niobate, comprising: an electrical control module 9 and a computer 10. The electrical control module 9 mainly implements the control of digital-to-analog conversion, analog-to-digital conversion, and modulator bias voltage; the computer 10 mainly trains the weights and processes digital signals.

[0056] Please see Figure 3 The present invention discloses a high-dimensional tensor convolution accelerator based on thin-film lithium niobate, wherein the information to be processed includes high-dimensional tensor data of images or videos. Loading the information to be processed specifically includes: firstly, separating the image tensor information into three-channel matrix data of R, G, and B; then flattening the separated three-channel matrix data into vectors, and loading the three-channel data onto three modulators in the first thin-film lithium niobate electro-optic intensity modulator array 2 respectively.

[0057] Optionally, the information to be processed includes high-dimensional tensor data of the video. Loading the information to be processed specifically includes: firstly, separating the video tensor information into four-channel matrix data of R, G, B, and t; then flattening the separated four-channel matrix data into vectors, and loading the four-channel data onto the four modulators in the first thin-film lithium niobate electro-optic intensity modulator array 2 respectively.

[0058] Optionally, the number of images or videos loaded onto the first thin-film lithium niobate electro-optic intensity modulator array 2 for parallel convolution calculation is greater than or equal to 1.

[0059] Specifically, in the second thin-film lithium niobate electro-optic intensity modulator array 5, for image convolution calculation, every three modulators form a convolution kernel, and each modulator loads the weight information of a single channel; for video convolution calculation, every four modulators form a convolution kernel, and each modulator loads the weight information of a single channel; different numbers of convolution kernels are set according to different processing tasks, and the number of convolution kernels is greater than or equal to 1.

[0060] Optionally, loading the weight information specifically involves: flattening the weight information of a single channel in the convolution kernel into a vector, and then loading the weight data of the single channel into each modulator in the second thin-film lithium niobate electro-optic intensity modulator array 5.

[0061] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A high-dimensional tensor convolution accelerator based on thin-film lithium niobate, characterized in that, The high-dimensional tensor convolution accelerator includes a sequentially connected array of light sources (1), a first thin-film lithium niobate electro-optic intensity modulator array (2), a wavelength division multiplexer (3), a beam splitter (4), a second thin-film lithium niobate electro-optic intensity modulator array (5), a wavelength division multiplexer array (6), a balanced detector array (7), and a charge integrator array (8). in, The light source array (1) is used to generate multiple optical carrier signals of different wavelengths; The first thin-film lithium niobate electro-optic intensity modulator array (2) is used to load the information to be processed and to modulate the optical carrier signal according to the loaded information to be processed to obtain an optical intensity signal carrying the information to be processed. The wavelength division multiplexer (3) is used for multiplexing multiple single-wavelength optical carriers; The beam splitter (4) is used to evenly distribute the optical carrier power; The second thin-film lithium niobate electro-optic intensity modulator array (5) is used to load weight information, modulate the light intensity signal output by the first thin-film lithium niobate electro-optic intensity modulator array (2) according to the loaded weight information, and combine with the balanced detector (7) to encode and detect negative weights, thereby obtaining an optical carrier signal carrying the information to be processed and the weight information. The wavelength demultiplexer array (6) is used for multi-wavelength optical carrier demultiplexing; The balance detector (7) is used to detect the calculation results of positive or negative weights; The charge integrator (8) is used to accumulate the signal detected by the detector.

2. The high-dimensional tensor convolution accelerator based on thin-film lithium niobate according to claim 1, characterized in that, The number of light sources N in the light source array (1) is a positive integer greater than or equal to 3; and it is integrated on a thin-film lithium niobate chip using a heterogeneous integration process.

3. A high-dimensional tensor convolution accelerator based on thin-film lithium niobate according to claim 1, characterized in that, The first thin-film lithium niobate electro-optic intensity modulator array (2) uses a 1×2 multimode interference coupler (MMI) to split and combine the light from the two arms of the input modulator. The number of modulators in the first thin-film lithium niobate electro-optic intensity modulator array (2) is the same as the number of light sources, and it is a push-pull Mach-Zehnder electro-optic intensity modulator.

4. A high-dimensional tensor convolution accelerator based on thin-film lithium niobate according to claim 1, characterized in that, The wavelength division multiplexer (3) has one N×1 wavelength division multiplexer, where N is a positive integer greater than or equal to 3; the wavelength demultiplexer array (6) has 2M 1×N wavelength demultiplexers, where M is a positive integer greater than or equal to 3.

5. A high-dimensional tensor convolution accelerator based on thin-film lithium niobate according to claim 1, characterized in that, The second thin-film lithium niobate electro-optic intensity modulator array (5) is a push-pull Mach-Zehnder electro-optic intensity modulator containing M modulators, where M is a positive integer greater than or equal to 3; the light from the two arms of the input modulator is split using a Y-branch beam splitter. A 2×2 MMI is used to interfere the two beams of light passing through the modulation arm, and finally the light is output from the two ports of the MMI.

6. A high-dimensional tensor convolution accelerator based on thin-film lithium niobate according to claim 1, characterized in that, The balanced detector array (7) consists of N×M units and is integrated on a thin-film lithium niobate chip using a flip-chip bonding process. M is a positive integer greater than or equal to 3, and N is a positive integer greater than or equal to 3.

7. A high-dimensional tensor convolution accelerator based on thin-film lithium niobate according to claim 1, characterized in that, The charge integrator array (8) consists of N×M units, where M is a positive integer greater than or equal to 3 and N is a positive integer greater than or equal to 3. It is integrated on a thin-film lithium niobate chip using heterogeneous integration technology.

8. A high-dimensional tensor convolution accelerator based on thin-film lithium niobate according to claim 1, characterized in that, The information to be processed includes high-dimensional tensor image or video data. Loading the information to be processed specifically includes: firstly, separating the image tensor information into R, G, and B three-channel matrix data; then flattening the separated three-channel matrix data into vectors, and loading the three-channel data onto the three modulators in the first thin-film lithium niobate electro-optic intensity modulator array (2); Alternatively: First, the video tensor information is separated into four-channel matrix data of R, G, B, and t; then the separated four-channel matrix data is flattened into vectors, and the four-channel data is loaded onto the four modulators in the first thin-film lithium niobate electro-optic intensity modulator array (2); where R, G, B, and t represent red, green, blue, and time, respectively.

9. A high-dimensional tensor convolution accelerator based on thin-film lithium niobate according to claim 1, characterized in that, In the second thin-film lithium niobate electro-optic intensity modulator array (5), for image convolution calculation, every three modulators form a convolution kernel, and each modulator loads the weight information of a single channel; for video convolution calculation, every four modulators form a convolution kernel, and each modulator loads the weight information of a single channel; different numbers of convolution kernels are set according to different processing tasks, and the number of convolution kernels is greater than or equal to 1. The number of images or videos loaded in the first thin-film lithium niobate electro-optic intensity modulator array (2) for image convolution calculation is greater than or equal to 1.

10. A high-dimensional tensor convolution accelerator based on thin-film lithium niobate according to claim 1, characterized in that, The loading of weight information specifically involves: flattening the weight information of a single channel in the convolution kernel into a vector, and then loading the weight data of a single channel into each modulator in the second thin-film lithium niobate electro-optic intensity modulator array (5).

Images