A method and system for photonic tensor convolution computation for multi-channel data processing

Abstract

The application discloses a kind of photonic tensor convolution calculation method and system for multi-channel data processing.The system includes multi-wavelength light source, high-speed modulator, signal generating unit, weighting unit, delay unit, balanced photodetector and signal processing unit, and the multi-channel tensor reconstruction algorithm of input signal.By improving the vectorization rule of tensor data, the elements of different channels in high-order tensor are rearranged into a one-dimensional vector.Based on the technology of three-dimensional multiplexing of time, space and frequency, only one modulator and a set of signal generating unit are needed to realize the convolution calculation of single optical link with arbitrary multi-channel tensor data simultaneously.The sliding convolution of multi-channel data can be directly completed in one calculation, which can release more physical hardware resources for multi-tensor parallel operation, complete more concise and efficient photonic tensor convolution operation, and greatly increase the integration and scalability of the scheme.

Description

TECHNICAL FIELD

[0001] The application belongs to the fields of photonic computing technology, multi-dimensional image processing technology and convolutional neural network technology, and particularly relates to a photonic tensor convolution computing method and system for multi-channel data processing for artificial intelligence. BACKGROUND

[0002] In the past few decades, the great development of artificial intelligence (AI) and Internet of Things (IoT) has driven the growing demand for high-performance computing (HPC) and real-time analysis of massive data. Tensor convolution computing is an effective mathematical method to extract the inherent structural features of multi-dimensional data, and plays a fundamental role in many fields such as autonomous driving, computer vision, biomedicine, and machine learning. Mainstream electronic processors convert multi-channel tensor convolution into general matrix multiplication (GeMM) to improve throughput and performance computing parallelism (i.e., channel-separated convolution), such as Tensor Core of Nvidia Ampere architecture. With the rise of AI and the slowing down of Moore's law, it is increasingly difficult to meet the computing power required by the explosive data flow using traditional electronic processors based on the Von Neumann architecture. Photonic computing technology, due to its inherent parallelism, ultra-high bandwidth, low latency and low energy consumption, has been proven to be a promising candidate for the next generation of neuromorphic computing in recent years, and is expected to solve the problem of wasted computing power and increased power consumption caused by the separation of memory and calculation and data dimension conversion in traditional electronic computing architecture. Existing photonic convolution technology, following the experience of electronic computing, converts standard tensor convolution operation (SC, Standard convolution) into general matrix multiplication operation (GEMM, General Matrix to Matrix Multiplication) according to the idea of depth separable convolution (see [Feldmann, J., Youngblood, N., Karpov, M. et al. Parallel convolutional processing using an integrated photonic tensor core. Nature 589, 52-58 (2021).]).

[0003] However, when the GEMM algorithm is used in the photonic convolution calculation architecture, each data patching needs a wavelength and a set of high-speed devices (including: electro-optical modulator, radio frequency amplifier and signal generator). Even after the optical time delay line patching, each convolution channel of the tensor needs a wavelength and a set of high-speed devices, and these high-speed radio frequency devices are expensive and difficult to integrate. Therefore, when processing large depth tensor data, the disadvantages of the photonic computing system using the GEMM algorithm gradually appear, and the low energy consumption advantage of optical computing can no longer be reflected. Therefore, we propose a method and system for photonic tensor convolution calculation suitable for processing arbitrary multi-channel data. SUMMARY

[0004] The technical problem to be solved by the present application is to overcome the complex optical architecture of the existing photonic convolution technology when processing multi-channel tensor data. Each convolution channel even each patching needs a wavelength and a set of high-speed devices (including: electro-optical modulator, radio frequency amplifier and signal generator). The present application improves the tensor data vectorization rule, rearranges the elements of different channels in the high-order tensor into a one-dimensional vector, and based on the technology of three-dimensional multiplexing of time, space and frequency, only one modulator and a set of signal generation units are needed to realize the convolution calculation of single optical link and arbitrary multi-channel tensor data at the same time, and the sliding convolution of multi-channel data can be directly completed in one calculation without clock synchronization between multiple channels. Therefore, more physical hardware resources can be released to perform multi-tensor parallel operation instead of channel parallel operation, and more concise and efficient photonic tensor convolution operation can be completed. Due to the reduction of the need for high-speed devices, the integrability and scalability of the scheme are greatly increased.

[0005] To achieve the above object, the present application provides the following technical scheme:

[0006] A photonic tensor convolution calculation system for multi-channel data processing, comprising a multi-wavelength light source (optical frequency comb or multi-wavelength laser array), a high-speed modulator, a signal generation unit, a weighting unit, a delay unit (dispersion delay or true time delay line), a balanced photodetector and a signal processing unit, and a multi-channel tensor reconstruction algorithm of input signal:

[0007] The multi-wavelength light source can generally be selected from an optical Kerr frequency comb, a semiconductor mode-locked laser or a semiconductor multi-wavelength laser array. It is used to generate stable multi-wavelength optical signals, and the information of the tensor convolution kernel and the tensor data information to be processed need to be loaded in the frequency domain and time domain, respectively. The more the number of wavelengths, the greater the amount of calculation at a time;

[0008] The high-speed modulator, because it needs to modulate multiple wavelengths simultaneously, generally uses a wideband Mach-Zehnder modulator (MZM), and the high-speed waveform generated by the tensor data is loaded on each wavelength simultaneously to complete the loading of the time domain information.

[0009] The signal generation unit is used for converting the original three-dimensional tensor data to be convolved into one-dimensional vector data according to the proposed algorithm, so as to facilitate loading on the optical signal through the high-speed radio frequency source and performing calculation on the optical system.

[0010] The weighting unit is used for loading the convolution kernel information in the frequency domain of each wavelength, and adjusting the output light signal intensity of each wavelength.

[0011] The delay unit is used for generating a bit of time delay between the time domain waveforms of each wavelength.

[0012] The balanced photodetector is used for loading the negative weight. The weight value of the convolution kernel is loaded on the corresponding wavelength, the size of the weight value corresponds to the size of the filtering power, and the positive and negative of the weight value corresponds to the corresponding port of the wavelength selection switch output. The positive value corresponds to the positive port of the balanced photodetector (BPD), and the negative value corresponds to the negative port of the BPD. Finally, the multi-wavelength modulated optical signal of the product accumulation operation is converted into an electrical signal, and the electrical signal is sent to the signal processing unit.

[0013] The signal processing unit is used for collecting the electrical signal output by the system after O / E conversion, and reconstructing the convolution feature image of the original picture through weighted mean filtering processing of the collected analog signal.

[0014] The multi-channel tensor reconstruction algorithm is used for reconstructing the original input multi-channel tensor into a one-dimensional vector array suitable for photonic convolution operation according to the improved im2col algorithm.

[0015] Further, the multi-wavelength light source used by the system can generally use a Kerr optical frequency comb, a semiconductor mode-locked laser, and a semiconductor laser array. Using the first two frequency combs, a single light source can obtain a larger number of wavelengths, but a wavelength selection switch needs to be used at the back end of the system to individually control each wavelength. Using a semiconductor laser array, because the intensity value of each wavelength is individually adjustable, there is no need for a wavelength selection switch,

[0016] Further, the time delay value (τ) of the multi-wavelength light source used by the system needs to match the modulation rate (B) of the system, that is, the dispersion value is one bit of the length of the time domain modulation waveform (τ=1 / B). In order to generate the required time delay value, the wavelength interval (Δλ) of the multi-wavelength light source needs to match the dispersion medium parameters (dispersion coefficient D and length L) or the length of the true time delay line, that is, τ=DLΔλ.

[0017] Further, the MZ modulator in the system is designed for wide optical bandwidth, so that more wavelengths can be modulated at the same time, a larger amount of calculation is completed, and the integrity of the optical signal is ensured. At the same time, the design and selection of low driving voltage reduces the use of radio frequency amplifiers, reduces the power consumption of the system, and ensures the integrity of the low frequency electrical signal.

[0018] Further, the weighting unit in the system can use a wavelength selection switch to control the intensity and routing of multiple wavelengths through a liquid crystal Lcos panel for a discrete system. For an integrated system on a chip, a micro ring resonator array can be used to adjust the resonant wavelength of each micro ring resonator through thermo-optic modulation, complete the customized intensity output of each wavelength, and correspond to the loading of the convolution kernel.

[0019] Further, the time delay unit in the system is generally loaded by a time delay unit through a dispersion time delay device or a true time delay line. The former is generally realized by a single-mode optical fiber with a certain dispersion coefficient, a chirped FBG grating, and a photonic crystal layer device. The latter is generally realized by a silicon on chip or a low-loss silicon nitride time delay line.

[0020] Further, the multi-channel tensor reconstruction algorithm includes the following steps:

[0021] S1, slice the original tensor according to the depth and convert it into d in two-dimensional m x m matrices.

[0022] S2, referring to the size (N x N x N) of the tensor convolution kernel, use a two-dimensional convolution kernel size of N x N, and expand the corresponding patching on d in channels into a one-dimensional array in sequence according to the im2col algorithm. The arrays obtained on the same patching and different channels are connected to form a longer one-dimensional array.

[0023] S3, according to the stride step size, generate a one-dimensional array connected in sequence between different channels for each corresponding convolution kernel patching in the input tensor.

[0024] S4, generate a one-dimensional array for each convolution kernel sliding patching position in the input tensor, and connect the first one in sequence according to the front and back order of the convolution kernel sliding, form the final optical one-dimensional array to be convolved, and broadcast it as a high-speed time domain modulation signal to each wavelength.

[0025] Further, the depth d in of the tensor to be processed can be of any size, and the tensor reconstruction algorithm is still effective. That is, a single device link can realize the standard convolution of tensors of any depth.

[0026] Further, in the reconstruction process of the to-be-processed tensor, column priority or row priority does not affect the result, efficiency and storage mode of the calculation.

[0027] A photon tensor convolution calculation method for multi-channel data processing, comprising the following steps:

[0028] S1. First, according to the proposed multi-channel tensor reconstruction algorithm, the original input multi-channel tensor is reconstructed into a one-dimensional vector array suitable for photon convolution operation.

[0029] S2. The integrated multi-wavelength light source generates an optical signal, and then N 3 wavelength signals are directly input into a broadband MZ modulator through end face coupling.

[0030] S2. The MZ modulator receives N 3 wavelengths, and the high-speed signals generated by the signal generator are amplified through a radio frequency amplifier and then modulated through the MZ modulator to obtain N 3 optical signals with the same high-speed time domain waveform, and are sent to a weighting unit.

[0031] S4. The weighting unit receives N 3 time-domain modulated optical signals, and adjusts the power of each wavelength according to the absolute value of the tensor convolution kernel to meet the corresponding value on the tensor convolution kernel. The wavelengths after weighting are sent to a time delay unit.

[0032] S5. The time delay unit receives N 3 wavelengths after time domain modulation and frequency domain weighting, and the dispersion unit generates a fixed time delay between the time domains of each wavelength, and then N 3 wavelengths are divided into two paths according to the positive and negative signs of the values on the convolution kernel and are sent to the two detection ports of the balanced photodetector, respectively.

[0033] S6. After the two detection ports of the balanced photodetector receive two signals, the optical signals are converted into electrical signals. Among them, the optical signals of the same path are accumulated in intensity in the photodetector, and the final electrical output signal is obtained through the differential processing of the balanced detector, which is the feature signal obtained after the tensor convolution operation of the to-be-convolved tensor signal.

[0034] Further, by integrating a Kerr comb, a semiconductor mode-locked laser or an integrated semiconductor laser array to generate multi-wavelength signals, more wavelengths can be provided for customized tensor convolution kernels.

[0035] Further, the MZ modulator is designed to have as large an optical bandwidth as possible to simultaneously modulate more wavelengths, ensuring the calculation amount and integration performance.

[0036] Further, since the general radio frequency amplifier is designed to pass high frequency and block low frequency in order to protect the high frequency circuit, signals of about 0-100 kHz are generally filtered out. Therefore, when the complex tensor signal is modulated, waveform distortion often occurs. Therefore, a high frequency carrier is added to the signal or a modulator with low driving voltage is used to ensure the integrity of the signal.

[0037] Further, the weighting unit uses a micro-ring resonator array or a semiconductor optical amplifier array. The former realizes the loading of the weight by controlling the resonant state of the micro-ring resonator through heat, and the latter realizes the loading of the weight by controlling the amplification and absorption of light through the particle number inversion state of the semiconductor PN junction.

[0038] Further, the time delay unit is loaded by a dispersion time delay device or a true time delay line. The former is generally realized by a single-mode optical fiber with a certain dispersion coefficient, a chirped FBG grating, and a photonic crystal layer device. The latter is generally realized by a silicon or silicon nitride time delay line on a chip.

[0039] Advantages of the present application:

[0040] 1) The present application is based on the high bandwidth, low energy consumption, and parallelism of photons, and simultaneously multiplexes the wavelength, time, and spatial dimensions of light to realize standard tensor calculation of multi-dimensional data, which can effectively avoid the increase in computational complexity and power consumption caused by the storage / computation separation and multi-dimensional data conversion in electronic calculation.

[0041] 2) The present application is based on a multi-channel tensor reconstruction algorithm, which rearranges the elements of different channels in a high-order tensor into a one-dimensional vector. Based on the three-dimensional multiplexing technology of time, space, and frequency, only one modulator and a set of signal generation units are needed to realize the convolution calculation of a single optical link and any multi-channel tensor data at the same time, and the sliding convolution of multi-channel data can be directly completed in one calculation. No additional high-speed optoelectronic functional devices are needed, so that the system can be simplified and the stability of the system can be improved, and the scalability of the scheme can be greatly increased.

[0042] 3) The present application is based on the way of realizing multi-channel tensor convolution by a single link, which does not need clock synchronization between multiple convolution channels. Therefore, more physical hardware resources can be released to perform multi-tensor parallel operation instead of single-tensor channel parallel operation, to complete more concise and efficient photonic tensor convolution operation.

[0043] The features and advantages of the present application will be described in detail in conjunction with the embodiments and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is a schematic diagram of the principle of the multi-channel tensor reconstruction algorithm of the present application;

[0045] Figure 2 A multi-channel tensor reconstruction algorithm step decomposition schematic diagram of the present application;

[0046] Figure 3 A multi-wavelength time domain sequence schematic diagram before tensor weighted time delay of the present application;

[0047] Figure 4 A multi-wavelength time domain sequence schematic diagram after tensor weighted time delay of the present application;

[0048] Figure 5 A tensor convolution process and its corresponding time domain misplacement accumulation schematic diagram of the present application;

[0049] Figure 6 A discrete system schematic diagram of the present application based on photon tensor convolution of multi-channel data processing;

[0050] Figure 7 An on-chip integrated system schematic diagram of the present application based on photon tensor convolution of multi-channel data processing;

[0051] Figure 8 A schematic diagram for an embodiment to complete 64-channel CT scan tensor data convolution processing by using a multi-layer convolutional neural network;

[0052] Figure 9 A data schematic diagram for an embodiment to complete 64-channel CT scan tensor data convolution processing by using a multi-layer convolutional neural network;

[0053] Figure 10 An effect schematic diagram for an embodiment to complete 64-channel CT scan tensor data feature extraction by using a multi-layer convolutional neural network;

[0054] Figure 11 An effect schematic diagram for an embodiment to complete 64-channel CT scan tensor data binary classification by using a multi-layer convolutional neural network; DETAILED DESCRIPTION

[0055] The present application proposes a photon tensor convolution calculation method and system for multi-channel data processing, solves the problems of high energy consumption and poor expansibility of a traditional photon tensor processing system caused by the dependence on a large number of high-speed radio frequency devices (high-speed modulator, high-speed signal generator, high-bandwidth radio frequency amplifier). In the proposed photon tensor convolution (PTPU) scheme, as a standard convolution calculation, multi-channel tensor convolution can be directly realized, and the joint mapping of feature space dimension and channel dimension is realized by using a tensor kernel. The loading of any multi-channel tensor data to be processed only needs one modulator.

[0056] The application first provides a photon tensor convolution calculation system for multi-channel data processing, which is composed of a multi-wavelength light source, a high-speed modulator, a signal generation unit, a weighting unit, a time delay unit, a balanced photodetector and a signal processing unit.

[0057] Firstly, according to the proposed multi-channel tensor reconstruction algorithm, the original input multi-channel tensor is reconstructed into a one-dimensional vector array suitable for photon convolution operation.

[0058] Then the integrated multi-wavelength light source generates optical signals, and then N 3 wavelength signals are directly input into a broadband MZ modulator through end face coupling. The MZ modulator receives N 3 wavelengths, and the high-speed signals generated by the signal generator are amplified through a radio frequency amplifier and then modulated by the MZ modulator, so that N 3 optical signals with the same high-speed time domain waveform are obtained and sent to the weighting unit.

[0059] The weighting unit receives N 3 time domain high-speed modulated optical signals, and adjusts the power of each wavelength according to the absolute value of the tensor convolution kernel to meet the corresponding value on the tensor convolution kernel. The weighted wavelengths are sent to the time delay unit.

[0060] The time delay unit receives N 3 wavelengths after time domain modulation and frequency domain weighting, and the dispersion unit generates a fixed time delay between the time domains of each wavelength, and then N 3 wavelengths are divided into two paths according to the positive and negative signs of the values on the convolution kernel and sent to the two detection ports of the balanced photodetector.

[0061] Finally, the two detection ports of the balanced photodetector receive two signals respectively, and convert the optical signals into electrical signals. Among them, the optical signals of the same path are accumulated in the photodetector, and the accumulated results are subtracted through the balanced detector to obtain the final electrical output signal, which is the feature signal obtained after the tensor convolution operation of the convolution tensor signal.

[0062] In this embodiment, the multi-wavelength light source used by the system can generally use Kerr optical frequency comb, semiconductor mode-locked laser, semiconductor laser array, etc. Using the first two frequency combs, a single light source can obtain a larger number of wavelengths, but a wavelength selection switch needs to be used at the back end of the system to individually control each wavelength. Using a semiconductor laser array, since the intensity value of each wavelength is individually adjustable, there is no need for a wavelength selection switch,

[0063] In this embodiment, the time delay value (τ) of the multi-wavelength light source used by the system needs to be matched with the modulation rate (B) of the system, that is, the dispersion value is the length of one bit of the time-domain modulation waveform (τ=1 / B). In order to generate the required time delay value, the wavelength interval (Δλ) of the multi-wavelength light source needs to be matched with the dispersion medium parameters (dispersion coefficient D and length L) or the length of the true time delay line, that is, τ=DLΔλ.

[0064] In this embodiment, the MZ modulator in the system is designed with a wide optical bandwidth, so that more wavelengths can be modulated at the same time, a larger amount of calculation can be completed, and the integrity of the optical signal is ensured. At the same time, low driving voltage is designed and selected, the use of radio frequency amplifiers is reduced, the power consumption of the system is reduced, and the integrity of the low-frequency electrical signal is ensured.

[0065] In this embodiment, the weighting unit in the system can use a wavelength selection switch to control the intensity and routing of multiple wavelengths through a liquid crystal Lcos panel for a discrete system. For an integrated system on a chip, a micro-ring resonator array can be used to adjust the resonant wavelength of each micro-ring resonator through thermal tuning, complete the customized intensity output of each wavelength, and correspond to the loading of the convolution kernel.

[0066] In this embodiment, the time delay unit in the system is generally loaded by a time delay unit through a dispersion time delay device or a true time delay line. The former is generally realized by a single-mode optical fiber with a certain dispersion coefficient, a chirped FBG grating, and a photonic crystal layer device. The latter is generally realized by a silicon on chip or a low-loss silicon nitride time delay line.

[0067] In this embodiment, the multi-channel tensor reconstruction algorithm includes the following steps:

[0068] S1, slice the original tensor according to the depth and convert it into d in two-dimensional m x m matrices.

[0069] S2, according to the size (N x N x N) of the tensor convolution kernel, use a two-dimensional convolution kernel size of N x N, and sequentially expand the corresponding patching on the d in channel into a one-dimensional array according to the im2col algorithm. The arrays obtained on the same patching and different channels are connected into a longer one-dimensional array.

[0070] S3, according to the stride step size, generate a one-dimensional array connected in sequence between different channels for each corresponding convolution kernel patching in the input tensor.

[0071] S4, generating a one-dimensional array for each convolution kernel in the input tensor corresponding to the patching position, sequentially connecting the first and the last according to the order of the convolution kernel sliding, forming the final one-dimensional array to be convolved in the light, and broadcasting it as a high-speed time domain modulation signal to each wavelength.

[0072] In this embodiment, the depth d of the tensor to be processed in Can be of any size, and the tensor reconstruction algorithm is still effective. That is, a single device link can realize the standard convolution of tensors of any depth.

[0073] In the reconstruction process of the tensor to be processed in this embodiment, column priority or row priority does not affect the calculation result, efficiency and storage mode.

[0074] The principle of the multi-channel tensor reconstruction algorithm proposed in the present application is as shown in Figure 1 、 2 ,

[0075] First, the original tensor to be processed is sliced according to the depth and converted into d in Two-dimensional m x m matrix.

[0076] Then, referring to the size (N x N x N) of the tensor convolution kernel, using the N x N two-dimensional convolution kernel size, the corresponding patching on the d in Channel is sequentially unfolded into a one-dimensional array according to the im2col algorithm. The arrays obtained on the same patching and different channels are connected into a longer one-dimensional array.

[0077] Next, according to the stride step size, each corresponding convolution kernel patching in the input tensor generates a one-dimensional array connected in turn between different channels.

[0078] Finally, the one-dimensional array generated by each convolution kernel in the input tensor corresponding to the patching position is sequentially connected in turn according to the order of the convolution kernel sliding, forming the final one-dimensional array to be convolved in the light, and broadcasting it as a high-speed time domain modulation signal to each wavelength.

[0079] For the standard tensor convolution operation, its formula can be expressed as:

[0080]

[0081] For the multi-channel tensor reconstruction algorithm proposed, its vectorization is:

[0082] X d，k = T d，Mod(k，N)，Fix(k / N) , k e (0, N 2 -1) (2)

[0083] After the photon convolution system, the product accumulation operation result is obtained:

[0084]

[0085] Remove some redundant information, and the effective data is:

[0086]

[0087] The multi-wavelength time-domain sequence before tensor weighting delay is as shown in Figure 3 After the vectorization signal of all wavelengths is loaded by the unified tensor, the time-domain waveforms are not misaligned and are completely aligned. Each intensity value in the frequency domain is filtered to a consistent level.

[0088] The multi-wavelength time-domain sequence after tensor weighting delay is as shown in Figure 4 After all wavelengths pass through the customized delay unit, uniform time delay misalignment occurs between each wavelength time domain, and is matched with the modulation rate (τ=1 / B=DLΔλ). In the frequency domain, according to the absolute value of the convolution kernel, the power of each wavelength is adjusted to the corresponding value to complete the mapping of the tensor convolution kernel. The multi-wavelength signals that have completed the frequency domain weighting and the time domain delay are detected by the photodetector to complete the product accumulation operation of each bit. As shown in the figure, the convolution of multiple channels can be directly completed in the operation again. Mark ① is the convolution result of the RG two channels, and mark ② is the convolution result of the GB two channels. As shown in Figure 5 The corresponding value of each bit is shown in detail.

[0089] The present application proposes a kind of for multi-channel data processing photonic tensor convolution calculation method and system, which is based on the discrete system of multi-channel data processing photonic tensor convolution as shown in Figure 6 Its running steps are as follows:

[0090] First, according to the multi-channel tensor reconstruction algorithm proposed above, the original input multi-channel tensor is reconstructed into a one-dimensional vector array suitable for photonic convolution operation.

[0091] Then, the integrated multi-wavelength light source generates optical signals, and then N 3 Wavelength signals are directly input into a broadband MZ modulator by end face coupling. The MZ modulator receives N 3 Wavelengths, and the high-speed signal generated by the signal generator is amplified by a radio frequency amplifier and then modulated by the MZ modulator to uniformly modulate these wavelengths, to obtain N 3 Optical signals with the same high-speed time-domain waveforms are sent to the weighting unit.

[0092] The weighting unit receives N 3The time domain modulated optical signal is power regulated according to the absolute value of the tensor convolution kernel for each wavelength, so as to meet the corresponding value on the tensor convolution kernel.

[0093] The time delay unit receives the N 3 wavelengths after time domain modulation and frequency domain weighting. 3 The dispersion unit generates a fixed time delay between the time domains of each wavelength, and then the N

[0094] Finally, the two detection ports of the balanced photodetector receive two signals respectively, and then convert the optical signal into an electrical signal.

[0095] The present application provides a kind of for multi-channel data processing photonic tensor convolution calculation method and system, which is based on multi-channel data processing photonic tensor convolution on-chip integrated system as shown in figure Figure 7 Different from the former, the weighted time delay unit is realized by micro-ring array and true time delay line respectively. The resonant state of the micro-ring resonator is controlled by heat to realize wavelength power modulation and complete the required weight loading.

[0096] As shown in figure Figure 8 A multi-layer convolutional neural network for completing multi-channel tensor convolution is established, which is used to verify the photonic tensor convolution calculation method and system proposed by the present application. Since the tensor used is a 64-channel CT scan image, it is vectorized and input into the photonic convolution system. The data obtained by the first layer convolution is shown in figure Figure 9 The enlarged view shows 63 different sampling points, which correspond to 63 convolution feature channels obtained by directly convolving the 64-channel tensor with a 2x2x2 convolution kernel, and also reflects the greatest advantage of the scheme, that is, direct convolution of any multi-channel tensor through a single optical link.

[0097] As shown in figure Figure 10 The feature extraction effect diagram of the photonic tensor convolution system is recovered by the sampling points in the above data (only the first thirty features are demonstrated). The feature extraction effect shows complete lower edge information, and the extraction effect is good.

[0098] In summary, the application discloses a photon tensor convolution calculation method and system suitable for processing of arbitrary multi-channel high-dimensional data, and is suitable for application scenarios such as machine vision, medical imaging, automatic driving and all deep learning networks containing tensor convolution operation. The technical problem to be solved by the application is that: in the prior art, when processing multi-channel tensors, according to the experience of electronic calculation, the standard tensor convolution operation (SC, Standard convolution) is converted into general matrix multiplication operation (GEMM, General Matrix to Matrix Multiplication) according to the idea of depth separable convolution. However, when the GEMM algorithm is used in the photon convolution calculation architecture, each data patching needs a wavelength and a set of high-speed devices (including: electro-optical modulator, radio frequency amplifier and signal generator). Even after the optical time delay line patching, each convolution channel of the tensor needs a wavelength and a set of high-speed devices, and these high-speed radio frequency devices are expensive and difficult to integrate. Therefore, when processing large depth tensor data, the disadvantages of the photon computing system using the GEMM algorithm will gradually appear, and the advantage of low energy consumption of optical computing can no longer be reflected. The application improves the tensor data vectorization rule, rearranges the elements of different channels in the high-order tensor into a one-dimensional vector, and based on the three-dimensional multiplexing technology of time, space and frequency, only one modulator is needed to realize the standard convolution calculation of single optical link and arbitrary multi-channel tensor data at the same time. The sliding convolution of multi-channel data can be directly completed in one calculation, without the need for clock synchronization between multiple channels. Therefore, more physical hardware resources can be released to perform multi-tensor parallel operation instead of channel parallel operation, to complete more concise and efficient, low-power photon tensor convolution operation. The system architecture is composed of a multi-wavelength light source (optical frequency comb or multi-wavelength laser array), a Mach-Zehnder modulator (MZM), a signal generation unit, a weighting unit, a delay unit (dispersion delay or true time delay line), a balanced photodetector and a signal processing unit.

[0099] The above merely describes the preferred embodiment of the application and should not be used to limit the application. Any modification, equivalent replacement or improvement made within the spirit and principle of the application should be included in the protection scope of the application.

Claims

1. A photonic tensor convolution computing system for multi-channel data processing, characterized by: The system comprises a multi-wavelength light source, a high-speed modulator, a signal generating unit, a weighting unit, a delay unit, a balanced photodetector and a signal processing unit, and a multi-channel tensor reconstruction algorithm of an input signal: The multi-wavelength light source is used to generate a stable multi-wavelength light signal, and information of a tensor convolution kernel and information of tensor data to be processed are loaded on a frequency domain and a time domain of the multi-wavelength light signal respectively; The more the number of wavelengths is, the greater the amount of calculation in a single time is; The high-speed modulator is used to load a high-speed waveform generated by the tensor data on each wavelength simultaneously, and complete loading of time domain information; The signal generating unit is used to convert three-dimensional tensor data to be convolved into one-dimensional vector data according to the proposed algorithm, so as to load the one-dimensional vector data on the light signal by a high-speed radio frequency source, and perform calculation on an optical system; The weighting unit is used to load convolution kernel information on a frequency domain of each wavelength, and load the convolution kernel information by adjusting an output light signal intensity of each wavelength; The delay unit is used to generate a bit of time delay between time domain waveforms of each wavelength; The balanced photodetector is used to load a negative weight value; the weight value of the convolution kernel is loaded on a corresponding wavelength, a size of the weight value corresponds to a filter power, and a positive or negative of the weight value corresponds to a corresponding port of a wavelength selection switch output; a positive wavelength accesses a positive port of the balanced photodetector, and a negative wavelength accesses a negative port of the balanced photodetector; finally, a multi-wavelength modulated light signal of a product accumulation operation is converted into an electrical signal, and the electrical signal is sent to the signal processing unit; The signal processing unit is used to collect an electrical signal output by the system after O / E conversion, and reconstruct a convolution feature image of an original picture by performing weighted mean filtering processing on the collected analog signal. The multi-channel tensor reconstruction algorithm is used to reconstruct a one-dimensional vector array suitable for photonic convolution operation according to an improved im2col algorithm.

2. The photonic tensor convolution calculation system for multi-channel data processing according to claim 1, wherein: The multi-wavelength light source used in the system comprises a Kerr optical frequency comb, a semiconductor mode-locked laser and a semiconductor laser array; the first two frequency combs are used to obtain a larger number of wavelengths by using a single light source, and a wavelength selection switch is used to individually control each wavelength at the rear end of the system; The semiconductor laser array is used, and the intensity value of each wavelength is individually adjustable, so that the wavelength selection switch is not needed; The time delay value τ of the multi-wavelength light source used in the system matches the modulation rate B of the system, that is, the dispersion value is one bit of length of the time domain modulation waveform, τ = 1 / B; in order to generate the required time delay value, the wavelength interval Δλ of the multi-wavelength light source matches the parameter of the dispersion medium or the length of the true time delay line, that is, τ = DLΔλ.

3. A photonic tensor convolution computing system for multi-lane data processing as defined in claim 1, wherein: The high-speed modulator in the system is designed to have a wide optical bandwidth, so as to simultaneously modulate more wavelengths, complete a larger amount of calculation, ensure the integrity of the light signal, simultaneously select a low driving voltage, reduce the use of a radio frequency amplifier, reduce the power consumption of the system and ensure the integrity of a low-frequency electrical signal.

4. A photonic tensor convolution computing system for multi-channel data processing as defined in claim 1, wherein: The weighting unit in the system uses a wavelength selective switch to control the intensity and routing of multiple wavelengths through a liquid crystal Lcos panel for a discrete system; for an on-chip integrated system, a micro ring resonator array is used to adjust the resonant wavelength of each micro ring resonator through thermo-optic modulation, complete the customized intensity output of each wavelength, and correspond to the loading of the convolution kernel.

5. A photonic tensor convolution computing system for multi-lane data processing as defined in claim 1, wherein: The time delay unit in the system is loaded through a dispersive time delay device or a true time delay line, the former is realized through a single-mode optical fiber with a predetermined dispersion coefficient, a chirped FBG grating and a photonic crystal layer device; the latter is realized through a silicon on-chip or a low-loss silicon nitride time delay line.

6. A photonic tensor convolution computing system for multi-lane data processing as defined in claim 1, wherein: The multi-channel tensor reconstruction algorithm includes the following steps: S1, slice the original tensor by depth, convert into d in two-dimensional m x m matrix; wherein the depth d in of the tensor to be processed is of any size, the tensor reconstruction algorithm is still effective, that is, the standard convolution of tensor of any depth is realized through a single device link. S2, the size of the reference tensor convolution kernel N x N x N, using the two-dimensional convolution kernel size N x N, d in The corresponding patching on the d channels is unfolded into a one-dimensional array in turn according to the im2col algorithm; the arrays obtained on different channels of the same patching are connected into a longer one-dimensional array; during the reconstruction of the tensor to be processed, column priority or row priority does not affect the calculation result, efficiency and storage mode; S3, according to the stride step size, the patching of each corresponding convolution kernel in the input tensor generates a one-dimensional array connected in turn between different channels; S4, the one-dimensional array generated by the patching position of each convolution kernel sliding in the input tensor is connected in turn according to the front and rear order of the convolution kernel sliding, forming the final one-dimensional array to be convolved on light, and broadcasting it as a high-speed time domain modulation signal to each wavelength.

7. A photonic tensor convolution computing method for multi-channel data processing based on the system of any one of claims 1-6, characterized in that, Including the following steps: S1, first, according to the proposed multi-channel tensor reconstruction algorithm, the original input multi-channel tensor is reconstructed into a one-dimensional vector array suitable for photonic convolution operation; S2, the integrated multi-wavelength light source generates optical signals, and then inputs N 3 wavelength signals into a broadband MZ modulator directly through end face coupling; S3, the MZ modulator receives N 3 wavelengths, the high-speed signal generated by the signal generator is amplified by the radio frequency amplifier, and then the MZ modulator uniformly modulates these wavelengths to obtain N 3 optical signals with the same high-speed time domain waveform, and sends them to the weighting unit; S4, the weighting unit receives N 3 time domain high-speed modulated optical signals, and performs power regulation on each wavelength according to the absolute value of the tensor convolution kernel so as to meet the corresponding value on the tensor convolution kernel; and the weighted wavelength is sent to the time delay unit. S5, the time delay unit receives N wavelengths after time domain modulation and frequency domain weighting, the dispersion unit generates a fixed time delay between the time domain of each wavelength, and then N wavelengths are divided into two paths according to the positive and negative signs of the numerical values on the convolution kernel and are respectively sent into the two detection ports of the balanced photodetector. 3 S5, the time delay unit receives N wavelengths after time domain modulation and frequency domain weighting, the dispersion unit generates a fixed time delay between the time domain of each wavelength, and then N wavelengths are divided into two paths according to the positive and negative signs of the numerical values on the convolution kernel and are respectively sent into the two detection ports of the balanced photodetector. 3 S5, the time delay unit receives N wavelengths after time domain modulation and frequency domain weighting, the dispersion unit generates a S6, after the two detection ports of the balanced photodetector receive two signals respectively, the optical signal is converted into an electrical signal; wherein the optical signals of the same channel are accumulated in the photodetector, and the accumulated result is processed by the balanced detector to obtain the final electrical output signal, which is the feature signal obtained after the tensor convolution operation of the convolution tensor signal.

8. The photonic tensor convolution calculation method for multi-channel data processing of claim 7, wherein: A high-frequency carrier is added to the signal or a low-drive-voltage modulator is used to ensure the integrity of the signal.

9. The photonic tensor convolution calculation method for multi-channel data processing of claim 7, wherein: The weighting unit uses a micro ring resonator array or a semiconductor optical amplifier array, the former realizes the loading of the weight by controlling the resonant state of the micro ring resonator through thermal modulation, and the latter realizes the loading of the weight by controlling the amplification and absorption of light through the particle number inversion state of the semiconductor PN junction.

Images