A three-dimensional multi-channel photonic convolution acceleration method

Abstract

The application discloses a three-dimensional multi-channel photonic convolution acceleration method, which is suitable for a photonic convolutional neural network in an application scene requiring high speed and strong feature extraction capability. By using multiplexing of three dimensions of radio frequency, space and wavelength, parallel convolution operation of multiple convolution kernel combinations and multiple convolution kernels is realized, and a data preprocessing method for improving convolution operation efficiency is used, operation speed and feature extraction capability are improved by processing data into a sequence suitable for architecture operation. Further using the signal processing method, the anti-noise capability and the accuracy of the experimental system based on the architecture are improved by optimizing the phase of each frequency component in the superimposed signal. The three-dimensional multi-channel photonic convolution acceleration method provided by the application is mainly used for multi-channel convolution operation on the basis of high parallelism, has the characteristics of high speed, strong feature extraction capability and the like, and provides a design basis for a photonic convolutional neural network used for complex tasks.

Description

TECHNICAL FIELD

[0001] The patent relates to the field of photonic neural networks, in particular to an architecture design suitable for high-parallel multi-channel photonic convolution in complex scenarios. The patent provides a three-dimensional multi-channel photonic convolution structure, which has the characteristics of high-parallel computing and flexible multi-channel, and can promote the development and application of photonic convolutional neural network technology. BACKGROUND

[0002] Photonic convolutional neural network technology is to realize convolutional neural network using optical method. Since light itself has multiple dimensions that can be multiplexed and has the characteristics of transmission and calculation, it can significantly improve the parallelism of calculation and reduce the energy consumption of operation. At the same time, convolutional neural network has strong feature extraction capability, so it can significantly improve the recognition ability of neural network. In recent years, photonic single-channel convolutional neural network has developed rapidly, and significant progress has been made in the research of convolution parallelism and integration. Photonic convolutional neural network is the main technology to improve computing power and small application of artificial intelligence terminal in the future. By multiplexing multiple optical dimensions to participate in convolution operation, the parallelism of convolution operation is greatly improved, and the calculation speed is improved. Through the integration of optical devices, the size of the system is reduced, the energy consumption cost is reduced, and it is easy to apply. However, due to the increasing complexity of current application scenarios, traditional single-channel photonic convolutional neural network cannot meet the demand of feature extraction capability of application. Therefore, in order to fully exert the advantages of photonic convolutional neural network and apply it to practical tasks, it is urgent to study high-parallel multi-channel photonic convolution technology.

[0003] By increasing the number of multiplexing dimensions used and designing the corresponding architecture to increase the number of convolution channels and flexibility, the capacity and rate of processing data can be increased, and the performance of photonic convolutional neural network can be significantly improved, and the practicality of photonic convolutional neural network can be promoted. The existing convolution architecture design method only considers high-dimensional multiplexing or multi-channel convolution on one side, and does not jointly design high-parallel and multi-channel convolution.

[0004] The joint design of high-parallel and multi-channel photonic convolutional neural network is a method of allocating multiple multiplexing dimensions to meet the parallelism of data and the flexible multiplexing of convolution channels. In order to maximize the utilization of computing performance, it is required that the physical quantities representing each parameter are relatively independent and have high utilization, that is, the adjustment of the data to be convolved does not affect the size of the convolution kernel and the number of channels, and at the same time, the adjustment of the number of convolution channels does not affect the data to be convolved participating in convolution.

[0005] The current actual application requires increasing speed and latency of data convolution processing, so on the basis of high-parallel and multi-channel convolution, it is required to have the computing capability of performing complete convolution operation in one clock cycle, that is, real-time processing of data.

[0006] In summary, in the complex application scenario of large data volume and strong feature extraction capability, how to design a photonic convolution unit architecture with high computational parallelism and multi-channel is the main problem to be solved. SUMMARY

[0007] The present patent mainly proposes a three-dimensional multi-channel photonic convolution acceleration method for the application scenario of photonic convolution neural network with strong feature extraction capability. It mainly considers how to perform convolution calculation with high parallelism and how to set up flexible multi-channel to achieve the goal of improving operation speed and feature extraction capability. It contains three technical points:

[0008] 1. A three-dimensional multi-channel photonic convolution unit architecture is proposed;

[0009] 2. A data preprocessing method for improving the efficiency of convolution operation is designed;

[0010] 3. A signal processing method for improving the accuracy of the experiment is proposed.

[0011] The first technical point is described in detail as follows:

[0012] A three-dimensional multi-channel photonic convolution unit architecture is proposed as shown in Figure 1 The architecture mainly consists of four parts, namely the radio frequency modulation part of the data to be convolved, the multi-wavelength optical signal generation part, the modulator modulation part, and the coupling demultiplexing detection part. Assuming that N different frequency radio signals, KxK modulators and M different wavelengths are used, NXM convolution operations can be completed in one clock cycle through the combination of the four parts.

[0013] In the radio frequency modulation part of the data to be convolved, part A in Figure 1 , the same N different frequency radio signals are used at each modulator, where the radio signals at the same modulator carry the same convolution position in N different convolution combinations, that is, the data to be convolved at the same position in N different convolution combinations is modulated onto the amplitude of the N different frequency radio signals at the same modulator. The signals are superimposed into a combined signal containing information of each frequency component, which can be converted from time domain to frequency domain by Fourier transform to extract the information. Therefore, the number of different frequency radio signals used at the same modulator represents the number of convolution combinations of the data to be convolved that can be operated in one clock cycle.

[0014] In the multi-wavelength optical signal generation part, corresponding to Figure 1In the middle C part, the tunable laser emits M different wavelength optical signals, where different wavelengths represent different convolution kernels, i.e., different convolution channels, and the power of the wavelength represents the weight value of the convolution kernel. The input of the same modulator represents the optical signal at the same convolution position in different convolution kernels. There are optical signals with the same wavelength but different powers at each modulator, where the same wavelength distribution but different powers correspond to different weight values at different convolution positions. Therefore, the number of wavelengths of the optical signals used represents the number of convolution channels that can be operated in a clock cycle.

[0015] In the modulator modulation part, the input optical signal is modulated by the modulator corresponding to Figure 1 In the B part of the modulator, the number of modulators is the same as the size of the convolution kernel. The superimposed radio frequency signal modulates the input optical signal through the modulator. The position of the convolution kernel weight value represented by the radio frequency signal at the same modulator and the optical signal represents the corresponding position. Therefore, the modulation process corresponds to the point multiplication operation in the corresponding position element in the convolution calculation. Therefore, through the action of K×K modulators, the multiplication operation of point multiplication in N×M convolution operations can be completed in a clock cycle, i.e., NMK 2 times of calculation operations.

[0016] In the coupling demultiplexing detection part, the input optical signal is coupled into one optical signal through the coupling part corresponding to Figure 1 In the D part of the coupling part, the coupling part couples the multiple optical signals into one optical signal, compensates for the insertion loss in the optical path through the doped fiber amplifier, demultiplexes the optical signals of different wavelengths through the demultiplexer, and obtains the final result through the photodetector. In the coupling process, the coupler couples the output optical signals of all modulators, i.e., sums the powers of optical signals with the same wavelength, and completes the summation operation of all positions in the convolution. Therefore, through the coupling action of the coupler, the summation part of point multiplication in N×M convolution operations can be completed in a clock cycle, i.e., NMK 2 times of calculation operations.

[0017] The proposed architecture can perform high-parallel multi-channel photonic convolution calculation, and can simultaneously perform convolution operation on N convolution data groups and M different convolution channels in a clock cycle, i.e., 2NMK 2 times of calculation operations.

[0018] The second technical point is described as follows:

[0019] A data preprocessing method for improving the efficiency of convolution operation is proposed, as shown in Figure 2 and Figure 3 Figure 2 ​The shown is the preliminary classification of the data to be convolved according to the different convolution operation positions, using a window with the same size as the convolution kernel to move in the data to be convolved with a step size of 1, first the window moves to the right, and after moving to the edge, it starts again from the leftmost side of the next row, the step size of the downward movement is still 1, and the elements are placed in different lists according to the position of the single data to be processed in the window, and the window moves according to the above rules until all the data to be convolved is traversed, that is, all the data to be convolved is placed in the list, and the number of lists is the size of the convolution kernel. Figure 3 The shown is the truncation of the list data according to the different frequency of the radio frequency signal, and finally a time sequence signal is formed. Therefore, by preprocessing the data to be convolved, it is adapted to the processing process of the architecture, so as to improve the efficiency of the convolution operation.

[0020] The third technical point is described in detail as follows:

[0021] A signal processing method for improving experimental accuracy is proposed. When multiple radio frequency signals of different frequencies are superimposed and summed, the phase of each frequency signal will affect the final signal amplitude. By optimizing the phase of all frequency component signals, the specific optimization method is to minimize the maximum absolute value of the superimposed signal by using the second-order optimization method Broyden-Fletcher-Goldfarb-Shanno (BFGS), wherein the maximum absolute value of the superimposed signal amplitude represents the function and the objective function of the phase optimization is:

[0022]

[0023] Where φ i is the phase of a single radio frequency signal, f i is the frequency of a single radio frequency signal; through the optimization of the phase, the peak value of the superimposed signal is reduced, the signal flatness is increased, the near-0 value in the time domain is reduced, and the anti-noise performance is enhanced. The processing capacity of the architecture for large-scale data and the robustness of complex data are improved, so as to achieve the purpose of improving the system calculation accuracy and application accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 It is a schematic diagram of a three-dimensional multi-channel photon convolution unit architecture. The arrows in the figure represent the signal flow direction of the radio frequency superimposed signal and the different wavelength optical signals. RF: radio frequency signal; λ: optical signal; MOD: modulator; TL: tunable laser; EDFA: erbium-doped fiber amplifier; PD: photodetector.

[0025] Figure 2 It is a schematic diagram of the data grouping process in a data preprocessing method for improving the efficiency of convolution operation.

[0026] Figure 3 A schematic diagram of data recombination in a data preprocessing method for improving the efficiency of convolution operation. The data is reorganized into a sequence suitable for architecture processing, that is, the architecture convolution operation ability in an adaptive clock cycle. N: the number of different frequency radio frequency signals; KxK: the number of modulators. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical method and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings, wherein: Figure 1 , accompanying Figure 2 , accompanying Figure 3 and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0028] In a specific embodiment of the three-dimensional multi-channel photonic convolution unit architecture proposed in the patent, different wavelengths of optical signals are used to represent different convolution kernels, which has stronger feature extraction capability compared with single-channel photonic convolution unit architecture.

[0029] The three-dimensional multi-channel photonic convolution process proposed in the architecture is as follows: Figure 1The first row at the top is taken as an example for illustration. The 40 to-be-convoluted data are loaded onto the amplitude values of the 40 radio frequency signals of different frequencies, wherein the 40 to-be-convoluted data are respectively data at the first position in 40 different convolution combinations. After phase optimization of all the radio frequency signals, the signals are superimposed, and the superimposed signals are subjected to electro-optical modulation by the first electro-optical modulator at the top. At this time, the powers of the two light signals of different wavelengths are respectively the weight values at the first position in the two convolution kernels, and therefore the modulation result is the point multiplication operation at the first position in the convolution operation of the 40 convolution data combinations on the two convolution kernels of size 2x2, that is, 40x2=80 times of multiplication operation is completed. Then, the operation is gradually extended to the other three rows. The amplitude values of the frequency components of the radio frequency signals in different modulators are different, and the powers of the two light signals of different wavelengths are different, which correspond to different position data in the original convolution combination and the convolution kernel respectively. The same operation is performed on the other three electro-optical modulators to complete all the point multiplication operations in the convolution operation. Through multiplexing of the 40 radio frequency signals, the four electro-optical modulators and the two wavelength light signals, 40x4x2=320 times of multiplication operation is completed in one cycle, the parallelism of the operation is significantly improved, and the operation of two convolution channels is performed at the same time, which significantly improves the feature extraction capability compared with the single-channel convolution architecture.

[0030] The output light signals of the four electro-optical modulators are coupled by a 4:1 coupler, and the powers of the light signals of the same wavelength in the four light signals are summed, that is, the powers of the light signals of 1550.12 nm and 1550.92 nm are summed. This step completes the operation of summing the point multiplication results at all positions in the convolution operation. After compensation of the insertion loss of the device by the doped optical fiber amplifier, the light signals of 1550.12 nm and 1550.92 nm are demultiplexed by the demultiplexer, and the powers of the light signals are detected by the photodetector to obtain the result.

[0031] The proposed data preprocessing method for improving the efficiency of convolution operation is as shown in Figure 2 and Figure 3As shown, assuming the size of the convolution kernel is 2x2, i.e. the window size in the data to be convolved is 2x2, first, from the top two rows, move to the right with a step size of 1, the data in the first position in the window, i.e. the data in the upper left corner, is placed in List1, the data in the second position, i.e. the data in the upper right corner, is placed in List2, the data in the third position, i.e. the data in the lower left corner, is placed in List3, and the data in the fourth position, i.e. the data in the lower right corner, is placed in List4. In the data reorganization process, it is assumed that 40 radio frequency signals of different frequencies are used, and the truncation with a length of 40, i.e. the reorganization into a matrix sequence data of 40 columns.

[0032] As can be seen from the above embodiments, the three-dimensional multi-channel photon convolution unit architecture proposed in the patent has the advantages of fast calculation speed brought by high parallel calculation, strong feature extraction capability brought by multi-channel convolution, etc.

Claims

1. A three-dimensional multi-channel photonic convolution acceleration method aims to realize multi-channel photonic convolution operation under high parallel operation capability by designing a three-dimensional multi-channel photonic convolution architecture and a data preprocessing method, and to realize the improvement of the calculation accuracy of the architecture by designing a signal processing method with anti-noise performance, and to promote the application process of photonic convolution neural network, which includes: A. High parallel operation is realized by a three-dimensional multi-channel photonic convolution unit architecture; the amplitude of the radio frequency signal is used to represent the data to be convolved, the number and power of the light signals of different wavelengths are used to represent the number and weight values of the convolution kernel, and the number of modulators is used to represent the size of the convolution kernel; the radio frequency signal representing the data to be convolved is modulated by the modulator representing the multi-channel convolution kernel, that is, the multiplication operation in the point multiplication operation, and the addition operation in the point multiplication is further realized by the coupling of the optical power, so as to realize the purpose of improving the parallel computing capability by using three-dimensional multiplexing technology; at the same time, the number of wavelengths is used to represent the number of channels, that is, the number of convolution kernels, wherein the number of wavelengths used represents the number of channels, and the power of the same wavelength light signal represents the weight value of the same convolution kernel; the number and power of the light signals can be flexibly adjusted according to the required number of channels and weight values, and the multi-channel convolution operation is realized by modulating and coupling the multi-wavelength composite light signal; B. The data preprocessing method is used to improve the convolution operation efficiency; according to the size of the convolution kernel and the number of radio frequency signals used, the data preprocessing method is used to classify the data to be convolved according to the convolution position by sliding the convolution window on the data to be convolved, and the data at the same position is placed in the same list, and the structure is further reorganized according to the convolution parameters of the architecture to adapt to the processing capacity of the architecture, so as to improve the convolution operation efficiency under the condition of large data amount; C. The signal processing method is used to improve the calculation accuracy; the optimization method of signal second-order amplitude balance is used to minimize the maximum absolute value of the superimposed signal by optimizing the phase relationship of the radio frequency signal, the signal flatness is increased, the near 0 value in the time domain is reduced, the anti-noise performance of the signal is enhanced, and the calculation accuracy of the architecture is improved.

2. The method of claim 1, wherein, High parallel operation is realized by three-dimensional multi-channel photon convolution unit architecture; the parallel degree of convolution calculation is improved by multiplexing in three dimensions of radio frequency, wavelength and space; the data to be convolved is modulated to the amplitude of N different frequency radio frequency signals to represent At the same time, K×K modulators are used to represent the size of the convolution kernel, i.e. multiplexing in space, and M different wavelength optical signals are used to represent the number of convolution channels, represented as The power of each wavelength optical signal when inputting different modulators represents different convolution weight values, i.e. a set of optical signals of the same wavelength on all modulators represents a complete convolution kernel; multi-channel convolution is realized by adjusting the number of convolution channels and the convolution kernel weight values; in the proposed architecture, the data to be convolved is first modulated to the amplitude of N different frequency radio frequency signals, and after signal superposition, it enters the modulator; the optical signal of a specific power of M different wavelengths is modulated, i.e. N×M point multiplication operations are completed in parallel, representing the operation of N data to be convolved and M convolution channels at a position; through the modulation of K×K modulators, the point multiplication operation is completed in one clock cycle; the summation operation is completed by coupling the modulated optical signal, i.e. the entire convolution operation is completed; the insertion loss in the optical path is compensated by amplification of the doped fiber amplifier; the optical signals of different wavelengths are demultiplexed by using a demultiplexer, and are detected by photoelectric detection respectively, and the continuous signals detected are processed by using Fourier transform to obtain the signal amplitude at each radio frequency, i.e. the convolution operation result.

3. The method of claim 1, wherein, The data preprocessing method is used to improve the convolution operation efficiency; a K×K window is set to move on the data to be convolved with a step of 1, the data in the window is classified according to the position in the window, and finally the data to be convolved is divided into K×K lists, the data in the same list represents the same position in different convolution operations, and corresponds to the same convolution weight value; the data is truncated according to the number of radio frequency signals used, that is, truncated in N columns, so that the method can adjust the data to be convolved to a time sequence with K×K rows and N columns, and realize the goal of improving the information processing efficiency of the architecture under the condition of large data amount.

4. The method of claim 1, wherein, The signal processing method is used to improve the calculation accuracy; after directly superimposing N radio frequency signals of different frequencies, the second-order Newton optimization algorithm is used to minimize the maximum absolute value of the superimposed signal, which is expressed as: （1） wherein is a phase of a single radio frequency signal, is a frequency of a single radio frequency signal; by optimizing the phase relationship of the radio frequency signals, the signal flatness is increased, the near 0 value in the time domain is reduced, the anti-noise performance of the superimposed signal is enhanced, and the purpose of improving the architecture calculation accuracy is achieved.

Images