Three-dimensional multichannel photon convolution acceleration method

Abstract

The invention discloses a three-dimensional multi-channel photon convolution acceleration method. The method is suitable for a photon convolution neural network in an application scene requiring high speed and strong feature extraction capability. Multiple to-be-convolved data combinations and parallel convolution operation of multiple convolution kernels are realized by using multiplexing of three dimensions of radio frequency, space and wavelength, and meanwhile, a data preprocessing method for improving convolution operation efficiency is utilized to process data into a sequence adaptive to architecture operation, so that the operation speed and the feature extraction capability are improved. Furthermore, the signal processing method is utilized to optimize the phase of each frequency component in the superposed signal, so that the anti-noise capability and the precision of an experiment system based on the architecture are improved. The three-dimensional multi-channel photon convolution acceleration method provided by the invention is mainly used for carrying out multi-channel convolution operation on the basis of high parallelism, has the characteristics of high speed, strong feature extraction capability and the like, and provides a design basis for a photon convolution neural network for complex tasks.

Description

Technical Field

[0001] This patent relates to the field of photonic neural networks, and in particular to an architecture design suitable for high-parallel multi-channel photonic convolution in complex scenarios. The patent provides a three-dimensional multi-channel photonic convolution structure, which has the characteristics of high parallel computing and flexible multi-channel, and can promote the development and application of photonic convolutional neural network technology. Background Art

[0002] Photonic convolutional neural network technology uses optical methods to realize convolutional neural networks. Since light itself has multiple dimensions that can be reused and has the characteristics of transmission and calculation, it can significantly improve the parallelism of calculations and reduce the energy consumption of calculations. At the same time, convolutional neural networks have strong feature extraction capabilities, so they can significantly improve the recognition ability of neural networks. In recent years, the development of photonic single-channel convolutional neural networks has become increasingly perfect, and research on the parallelism and integration of convolution has made significant progress. Photonic convolutional neural networks are the main technology for improving computing power and miniaturization of artificial intelligence terminals in the future. By reusing multiple optical dimensions to participate in convolution operations, the parallelism of convolution operations is greatly improved, thereby increasing the speed of calculations. By integrating optical devices, the size of the system is reduced, the energy consumption cost is reduced, and it is easy to apply. However, due to the increasing complexity of current application scenarios, traditional single-channel photonic convolutional neural networks can no longer meet the application's requirements for feature extraction capabilities. Therefore, in order to give full play to the advantages of photonic convolutional neural networks and apply them to practical tasks, it is urgent to study high-parallel multi-channel photonic convolution technology.

[0003] By increasing the reuse dimension used and designing the corresponding architecture to increase the number of channels and flexibility of convolution, the capacity and rate of data processing can be increased, the performance of photonic convolutional neural networks can be significantly improved, and the practical application of photonic convolutional neural networks can be promoted. Existing convolutional architecture design methods only consider high-dimensional reuse or multi-channel convolution, and have never combined high parallelism and multi-channel convolution for architecture design.

[0004] The highly parallel and multi-channel joint design of the photonic convolutional neural network is a method of allocating multiple reuse dimensions to meet the parallelism of data and the flexible reuse of convolution channels. In order to maximize the utilization of computing performance, the physical quantities that characterize each parameter are required to be relatively independent and have high utilization, that is, the adjustment of the convolution data does not affect the size of the convolution kernel and the number of channels, and the adjustment of the number of convolution channels does not affect the convolution data participating in the convolution.

[0005] Current practical applications have increasing requirements for the speed and latency of data convolution processing. Therefore, based on high parallelism and multi-channel convolution, it is necessary to have the computing power to perform a complete convolution operation within one clock cycle, that is, to process data in real time.

[0006] In summary, in complex application scenarios with large amounts of data and requiring strong feature extraction capabilities, how to design a photonic convolution unit architecture with high computational parallelism and multi-channels is the main problem that should be solved. Summary of the invention

[0007] This patent mainly proposes a three-dimensional multi-channel photon convolution acceleration method for application scenarios that require photon convolutional neural networks to have strong feature extraction capabilities. It mainly considers how to perform convolution calculations in high parallelism and how to perform flexible multi-channel settings to achieve the goal of improving computing speed and feature extraction capabilities. It contains three technical points:

[0008] 1. A three-dimensional multi-channel photon convolution unit architecture is proposed;

[0009] 2. Designed a data preprocessing method to improve the efficiency of convolution operation;

[0010] 3. A signal processing method is proposed to improve experimental accuracy.

[0011] The first technical point is described in detail as follows:

[0012] A three-dimensional multi-channel photon convolution unit architecture is proposed. Figure 1 As shown in the figure. The architecture mainly consists of four parts, namely the RF modulation part of the data to be convolved, the multi-wavelength optical signal generation part, the modulator modulation part and the coupled demultiplexing detection part. Assuming that N RF signals with different frequencies, K×K modulators and M different wavelengths are used, the combination of the four parts can complete N×M convolution operations in one clock cycle.

[0013] In the RF modulation part of the data to be convolved, corresponding to Figure 1 In part A, the same N RF signals of different frequencies are used at each modulator, where the RF signal at the same modulator carries the same convolution position in N different convolution combinations, that is, the data to be convolved at the same position in N different convolution combinations is modulated to the amplitude of N RF signals of different frequencies at the same modulator, and the signals are superimposed into a composite signal, which contains information of each frequency component. The time domain signal can be converted to the frequency domain through Fourier transform to extract this information. Therefore, the number of RF signals of different frequencies used at the same modulator represents the number of combinations of data to be convolved that can be calculated within one clock cycle.

[0014] In the generation part of multi-wavelength optical signal, corresponding Figure 1In the middle C part, the tunable laser emits M optical signals of different wavelengths, where different wavelengths represent different convolution kernels, i.e. different convolution channels, and the power of the wavelength represents the weight value of the convolution kernel. The input of the same modulator represents the optical signal at the same convolution position in different convolution kernels. At each modulator, there are optical signals with the same wavelength but different powers, where the wavelength distribution is the same but the power is different, corresponding to different weight values ​​at different convolution positions. Therefore, the number of wavelengths of the optical signal used represents the number of convolution channels that can be calculated within one clock cycle.

[0015] In the modulation part of the modulator, the corresponding Figure 1 In the B part, the number of modulators is the same as the size of the convolution kernel. The superimposed RF signal modulates the input optical signal through the modulator, where the positions of the data to be convolved represented by the RF signal at the same modulator and the convolution kernel weight value represented by the optical signal are corresponding, so the modulation process corresponds to the point multiplication operation of the corresponding position elements in the convolution calculation. Therefore, after the action of K×K modulators, the multiplication operation of the point multiplication in N×M convolution operations can be completed in one clock cycle, that is, NMK 2 calculation operations.

[0016] In the coupled demultiplexing detection part, corresponding to Figure 1 In the D part, the coupling part couples multiple optical signals into one optical signal, compensates for the insertion loss in the optical path through the erbium-doped fiber amplifier, demultiplexes the optical signals of different wavelengths through the demultiplexer, and uses the photodetector to detect them separately to obtain the final result. During the coupling process, the coupler couples the output optical signals of all modulators, that is, sums the power of the optical signals with the same wavelength, and completes the summation operation of all positions in the convolution. Therefore, through the coupling effect of the coupler, the summation part of the point multiplication of N×M convolution operations is completed in one clock cycle, that is, NMK 2 calculation operations.

[0017] The proposed architecture can perform highly parallel multi-channel photon convolution calculations. In one clock cycle, N convolution data combinations and M different convolution channels can be convolved at the same time, that is, 2NMK can be performed. 2 calculation operations.

[0018] The second technical point is specifically described as follows:

[0019] A data preprocessing method to improve the efficiency of convolution operation is proposed, such as Figure 2 and Figure 3 shown. Figure 2As shown in the figure, the data to be convolved are preliminarily classified according to the different positions of the convolution operation. A window with the same size as the convolution kernel is used to move in the data to be convolved with a step size of 1. First, the window moves to the right, and then starts again from the leftmost side of the next row after moving to the edge. The step size of moving downward is still 1. Inside the window, the elements are placed in different lists according to the different positions of the single data to be processed. The window is moved according to the above rules until all the data to be convolved are traversed, that is, all the data to be convolved are placed in the list, and the number of lists is the size of the convolution kernel. Figure 3 The table data is truncated according to the number of RF signals of different frequencies used, and finally a time series signal is formed. Therefore, by preprocessing the convolution data to adapt it to the processing process of the architecture, the purpose of improving the efficiency of the convolution operation can be achieved.

[0020] The third technical point is specifically described as follows:

[0021] A signal processing method to improve the experimental accuracy is proposed. When multiple RF signals of different frequencies are superimposed and summed, the phase of the signal of each frequency will affect the final signal amplitude. The phase of the signal of all frequency components is optimized. The specific optimization method is to use the second-order optimization method Broyden-Fletcher-Goldfarb-Shanno (BFGS) to minimize the maximum absolute value of the superimposed signal, where the maximum absolute value representation function of the superimposed signal amplitude and the objective function of the phase optimization are:

[0022]

[0023] where φ i is the phase of a single RF signal, f i is the frequency of a single RF signal; through phase optimization, the peak value of the superimposed signal is reduced, the signal flatness is increased, the near-zero value in the time domain is reduced, and the anti-noise performance is enhanced. The architecture's ability to handle large-scale data and its robustness to complex data are improved, thus achieving the goal of improving the system's calculation accuracy and application accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of a three-dimensional multi-channel photon convolution unit architecture. The arrows at the RF superposition signal and the optical signals of different wavelengths in the figure indicate the signal flow direction. RF: radio frequency signal; λ: optical signal; MOD: modulator; TL: tunable laser; EDFA: erbium-doped fiber amplifier; PD: photodetector.

[0025] Figure 2 A schematic diagram of the data grouping process in a data preprocessing method for improving the efficiency of convolution operations.

[0026] Figure 3 A schematic diagram of data reorganization in a data preprocessing method for improving the efficiency of convolution operations. The data is reorganized into a sequence that is adapted to the processing of the architecture, that is, adapted to the architecture convolution operation capability within one clock cycle. N: number of RF signals of different frequencies; K×K: number of modulators. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical method and advantages of the present invention more clear, the following Figure 1 , Attachment Figure 2 , Attachment Figure 3 It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0028] In a specific embodiment of a three-dimensional multi-channel photon convolution unit architecture proposed in this patent, light signals of different wavelengths are used to represent different convolution kernels, which has stronger feature extraction capabilities than a single-channel photon convolution unit architecture.

[0029] The three-dimensional multi-channel photon convolution process proposed in this architecture is as follows Figure 1As shown. Assume that 40 RF signals with different frequencies are used, and the frequencies are distributed at equal intervals of 10 MHz from 10 MHz to 400 MHz. Assume that the size of the convolution kernel is 2×2, that is, four electro-optic modulators are required for modulation. Assume that the wavelength of the optical signal emitted by the tunable laser is 2, which are 1550.12 nm and 1550.92 nm respectively, that is, the number of convolution channels is 2. For the convenience of introduction, first take the first path at the top as an example to illustrate, load 40 data to be convolved onto the amplitudes of RF signals of 40 different frequencies, where the 40 data to be convolved are the data of the first position in 40 different convolution combinations, and superimpose all RF signals after phase optimization. The superimposed signal is electro-optically modulated by the first electro-optical modulator at the top for the two wavelength signals emitted by the tunable laser. At this time, the powers of the two optical signals of different wavelengths are the weight values ​​of the first position in the two convolution kernels, respectively. Therefore, the modulation result is that 40 convolution data combinations simultaneously perform point multiplication operations on the first position in the convolution operation of two convolution kernels of size 2×2, that is, 40×2=80 multiplication operations are completed, and then gradually extended to the other 3 paths. The amplitudes of the frequency components of the RF signals in different modulators are different, and the powers of the optical signals of two different wavelengths are different, corresponding to the data at different positions in the original convolution combination and the convolution kernel, respectively. The same operation is applied to the other three electro-optical modulators to complete all point multiplication operations in the convolution operation. By multiplexing 40 RF signals, 4 electro-optical modulators and two wavelength optical signals, 40×4×2=320 multiplication operations are completed in one cycle, which significantly improves the parallelism of the operation. At the same time, operations on two convolution channels are performed, which significantly improves the feature extraction capability compared to the single-channel convolution architecture.

[0030] The 4 output optical signals of the 4 electro-optical modulators are coupled through a 4:1 coupler, and the power of the optical signals of the same wavelength in the 4 optical signals is summed, that is, the optical signals of 1550.12nm and 1550.92nm are summed. This step completes the operation of summing the point product results of each position in the convolution operation, and the insertion loss of the device is compensated by the erbium-doped fiber amplifier. Finally, the optical signals of 1550.12nm and 1550.92nm are demultiplexed by the demultiplexer and the power detection results are obtained using photodetectors.

[0031] A data preprocessing method is proposed to improve the efficiency of convolution operation. Figure 2 and Figure 3As shown, assuming that the size of the convolution kernel is 2×2, that is, the window size in the data to be convolved is 2×2, first start from the top two rows and move rightward with a step size of 1. The data at the first position in the window, i.e., the upper left corner, is placed in List1, the data at the second position, i.e., the upper right corner, is placed in List2, the data at the third position, i.e., the lower left corner, is placed in List3, and the data at the fourth position, i.e., the lower right corner, is placed in List4. In the data reorganization process, it is assumed that 40 RF signals of different frequencies are used, and they are truncated with a length of 40, i.e., reorganized into matrix sequence data of 40 columns.

[0032] It can be seen from the above embodiments that the three-dimensional multi-channel photon convolution unit architecture proposed in this patent has the advantages of fast computing speed brought by high parallel computing and strong feature extraction capability brought by multi-channel convolution.

Claims

1. A three-dimensional multi-channel photon convolution acceleration method aims to achieve multi-channel photon convolution operations under high parallel computing capabilities by designing a three-dimensional multi-channel photon convolution architecture and a data preprocessing method. At the same time, by designing a signal processing method with noise resistance, the accuracy of the experimental system based on the architecture is improved, and the application process of photon convolution neural network is promoted. The main contents include: A,three-dimensional multi-channel photon convolution unit architecture; By independently representing various parameters in the convolution process, that is, using the amplitude of the radio frequency signal to represent the data to be convolved, the number and power of optical signals of different wavelengths to represent the number and weight value of the convolution kernels, and the number of modulators to represent the size of the convolution kernel, the radio frequency signal representing the data to be convolved is used to modulate the optical signal representing the multi-channel convolution kernel using a modulator, that is, the multiplication operation in the point multiplication operation is performed, and the addition operation in the point multiplication is further performed on the superposition effect of the optical power by the coupler, so as to achieve the purpose of improving the parallel computing capability by using three-dimensional multiplexing technology; at the same time, by using rich and flexible wavelength resources to represent the number of convolution kernels, that is, the number of channels, the number of wavelengths used represents the number of channels, and the power of the optical signal of the same wavelength represents the weight value of the same convolution kernel, the number of wavelengths and the power of the optical signal can be flexibly adjusted according to the required number of channels and weight values, and the multi-channel convolution operation is achieved by modulating and coupling the multi-wavelength composite optical signal; B. A data preprocessing method for improving the efficiency of convolution operation; the data preprocessing method is performed according to the size of the convolution kernel and the number of radio frequency signals used, and the data to be convolved is classified according to the convolution position by sliding the convolution window on the data to be convolved, and the data at the same position is placed in the same list, and the structure is further reorganized according to the convolution parameters of the architecture to adapt to the processing capacity of the architecture, so as to achieve the goal of improving the efficiency of convolution operation in the case of large data volume; C. A signal processing method to improve experimental accuracy; By utilizing the optimization method of second-order amplitude equalization of signals and optimizing the phase relationship of RF signals to minimize the maximum absolute value of the superimposed signal, the signal flatness is increased and the near-zero values ​​in the time domain are reduced, thereby achieving the goal of enhancing the signal's anti-noise performance and improving the accuracy of the experimental system based on this architecture.

2. The method according to claim 1, characterized in that: A three-dimensional multi-channel photon convolution unit architecture design is proposed; the multiplexing of radio frequency, wavelength and space is used to improve the parallelism of convolution calculation; the convolution data is modulated onto the amplitude of N radio frequency signals of different frequencies represented as RF i (i=1,2,...,N), and K×K modulators are used to represent the size of the convolution kernel, that is, spatial multiplexing, and M optical signals of different wavelengths are used to represent the number of convolution channels, which is expressed as λ i (i=1,2,...,M), the power of each wavelength of optical signal at the input of different modulators represents different convolution weight values, that is, a set of optical signals of the same wavelength on all modulators represents a complete convolution kernel; utilizing the richness and flexibility of wavelength resources, the number of convolution channels and the convolution kernel weight values ​​can be flexibly adjusted to achieve the purpose of multi-channel convolution; in the proposed architecture, the data to be convolved is first modulated to the amplitude of N RF signals of different frequencies, and the signals are superimposed and enter the modulator to modulate the optical signals of M different wavelengths with specific powers, that is, N×M point multiplication operations are completed in parallel, indicating the operation of N data to be convolved and M convolution channels at one position, and NMK is completed within one clock cycle through the modulation of K×K modulators. 2 The first point multiplication operation is performed; the summation operation is completed by coupling the modulated optical signal, that is, the entire convolution operation is completed; the insertion loss in the optical path is compensated by amplification of the erbium-doped fiber amplifier; the optical signals of different wavelengths are demultiplexed by a demultiplexer, and photoelectric detection is performed separately, and the detected continuous signal is processed by Fourier transform to obtain the signal amplitude at each RF frequency, that is, the convolution operation result.

3. The method according to claims 1 and 2, characterized in that A data preprocessing method is proposed. A K×K window, i.e., the size of the convolution kernel, is set to move on the data to be convolved with a step size of 1. The data in the window is classified according to their position in the window. Finally, the data to be convolved is divided into K×K lists. The data in the same list have the same position in different convolution operations and correspond to the same convolution weight value. The data is truncated according to the number of RF signals used, i.e., it is truncated in units of N columns. Therefore, this method can adjust the convolution data to a time series with K×K rows and N columns as units by preprocessing the convolution data, thereby achieving the goal of improving the efficiency of architectural information processing under large data volume conditions.

4. The method according to claims 1 and 2, characterized in that A method for enhancing the signal anti-noise performance is proposed; after N RF signals of different frequencies are directly superimposed, the maximum absolute value of the superimposed signal amplitude is minimized by the second-order Newton optimization algorithm, which is expressed as: where φ i is the phase of a single RF signal, f i is the frequency of a single RF signal; by optimizing the phase relationship of the RF signal, the signal flatness is increased, the near-zero value in the time domain is reduced, and the anti-noise performance of the superimposed signal is enhanced, thereby achieving the purpose of improving the application accuracy of the architecture.

Images