A time, frequency and space three-dimensional interleaving photonic convolution accelerator for color image
By leveraging the power-dividing capabilities of channel shuffling and waveform shaping, combined with a hybrid photonic convolutional neural network, a highly efficient parallel convolution of the photonic convolutional accelerator in high-resolution color image processing was achieved. This solved the problem of insufficient spectrum resources and improved processing efficiency and accuracy.
Patent Information
- Application Number
- CN202310218905.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-09
- Publication Date
- 2026-02-27
- Estimated Expiration
- 2043-03-09
AI Technical Summary
Existing photonic convolution accelerators suffer from low single-channel convolution efficiency and insufficient utilization of spectral resources when processing high-resolution color images, making it difficult to efficiently process multi-channel data.
By employing a channel shuffling strategy and the power-dividing function of a waveform shaper, combined with an optoelectronic hybrid convolutional neural network, a photonic convolutional accelerator is constructed through time, frequency, and space three-dimensional interweaving. This enables random shuffling of multi-channel data and reuse of spectrum resources, constructing multiple photonic convolutional kernels and executing convolution operations in parallel.
It improves the processing efficiency and accuracy of photonic convolution accelerators, enabling efficient and rapid high-resolution color image classification with limited spectrum resources, while reducing hardware and energy consumption.
Smart Images
Figure CN116227567B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of photonic convolution accelerator, in particular to a time, frequency and space three-dimensional interlaced photonic convolution accelerator for color images. BACKGROUND
[0002] An artificial neural network is an operation model simulating the construction of a biological neural system. The model abstracts the operation mechanism of brain tissue, uses a large number of neurons as basic operation units, and connects them to form a complex network structure. By adjusting and optimizing network parameters, the model can learn and perform some prediction or judgment operations. Further development can also perform complex operations such as face recognition, autonomous driving and disease diagnosis.
[0003] However, the traditional neural network calculation process mostly uses electrical signals as information carriers. The storage and movement of data in the calculation process will consume a large amount of energy and have a limited operation time bottleneck due to the limitation of electronic clock. Optical calculation has the characteristics of low power consumption, light speed processing and parallel processing. If the information carrier in the operation process is changed to optical signal and optical calculation is used, some bottlenecks of electronic calculation can be overcome.
[0004] A photonic neural network can be regarded as an extension of optical calculation, which adopts an architecture combining storage and calculation, so as to improve the calculation speed while effectively reducing the calculation delay. The information carrier of the photonic neural network is light, which can ensure less loss in the network and reduce power consumption. Compared with traditional electronic integrated circuits, optical devices have larger bandwidth and shorter response time. In the calculation process, relying on the high-speed propagation of light, real-time high-speed calculation can be completed.
[0005] At present, parallel, high-speed and trainable photonic neural networks have made important progress, such as integrating the entire operation structure on a single photonic chip to achieve ultra-high calculation density. There are mainly three ways to realize optical neural networks.
[0006] The first implementation way is to realize through a Mach-Zehnder interferometer (MZI). The theoretical basis of this way is the triangular decomposition algorithm proposed by Reck et al. in 1994, which proves that MZI can realize any rotation matrix, and diagonal matrix is realized through an optical phase shifter. The combination of the two can realize a convolutional neural network at the experimental level, with the advantages of high integration and strong reconfigurability. At present, the processing dimension of the accelerator based on Mach-Zehnder interferometer is limited by the number of integrated ports, which leads to its inability to efficiently process high-resolution images; the weight of the accelerator based on the principle of spatial diffraction is difficult to control, which limits the flexibility of its complete function, in addition, it also has the problem of inaccurate weight mapping, so it is difficult to perform such complex task processing for high-resolution color pictures.
[0007] The second implementation is a phase mask-based diffractive photonic neural network, which can directly input optical signals compared to MZI, and has the possibility to implement large-scale and complex neural networks. The research in this aspect is mainly based on a diffractive photonic deep neural network structure proposed by Xing Lin et al. in 2018, which can complete the classification of handwritten digit dataset and Fashion dataset.
[0008] The third implementation is a photonic convolution accelerator proposed by Xingyuan Xu et al. in 2021, which applies the concept of time-wavelength multiplexing to optical neural networks and achieves a convolution operation of 11 billion times per second, completing the classification of handwritten digit dataset. However, when implementing multiple parallel convolution kernels, a large amount of spectral resources is required, and spectral resources are very scarce in communication and other fields. Therefore, it is necessary to implement efficient convolution acceleration of color images under the condition of occupying a small amount of spectral resources.
[0009] The information of a color picture often exists among multiple channels, and each channel has its own unique information. Due to the limitation of the design principle, the original photonic convolution accelerator can only perform convolution on single-channel data in one convolution operation. If different channels are convolved separately, the processing speed of a single picture will be reduced by three times. Another method is to build three sets of photonic convolution accelerators, which can ensure the processing speed of a single picture, but requires more hardware and energy resources. Therefore, it is necessary to develop a new convolution method to enable the photonic convolution accelerator to efficiently process multi-channel data. At the same time, in the original photonic convolution kernel construction scheme, the number of weights contained in the convolution kernel corresponds to the number of optical frequency combs required, and a large amount of spectral resources is required when completing multiple convolution kernel operations. Therefore, improving the utilization rate of limited spectral resources is an effective way to improve the efficiency of photonic convolution calculation. SUMMARY
[0010] The present application comprehensively considers the advantages and disadvantages of the current photonic convolution accelerator, and aims to propose a time, frequency and space three-dimensional interlaced photonic convolution accelerator for color images. The application continues the advantages of the photonic convolution accelerator based on separate photonic devices, such as fast convolution speed, ability to process large-scale pictures, modular construction, and scalable expansion. The application innovatively introduces a channel shuffling strategy in the picture input stage of the photonic convolution accelerator, which enables it to process color pictures. At the same time, under the condition of limited spectral resources, the application can construct multiple photonic convolution links in space, realize parallel convolution of the photonic convolution accelerator through repeated use of spectral resources, and multiply the output information amount of the photonic convolution layer. At the same time, the information of multiple feature dimensions is integrated and utilized, and finally the classification of high-resolution color images is completed efficiently and quickly.
[0011] In order to achieve the above object, the present application provides the following technical solutions:
[0012] The present application provides a color image-oriented time, frequency and space three-dimensional interleaving photonic convolution accelerator, comprising optical frequency comb signal, power amplifier, waveform shaper, coupler, optical spectrum analyzer, polarization controller, polarization beam splitter, electro-optic modulator, dispersion optical fiber, high-speed photodetector, AWG signal generator and oscilloscope, and the following processing steps: the high-resolution color picture is first processed by channel shuffling operation, and the single-channel picture data is reserved, and then is flattened into a one-dimensional vector and loaded onto the power of the time sequence electrical signal X, and the symbol rate is 1 / τ(baud), wherein τ is the symbol period; then a plurality of convolution kernels are respectively expanded into one-dimensional vectors, the optical frequency comb signal is divided into different output ports through the power division function of the waveform shaper, and then is shaped according to the weights of the corresponding convolution kernels, so that the corresponding weights of the plurality of convolution kernels are respectively encoded onto the power of the optical frequency comb signal of different output ports; then, the time sequence electrical signal X modulates the optical frequency comb signal through the electro-optic modulator to generate the product result of the weight vector W; then, through the dispersion optical fiber, an equal dispersion delay is superimposed on the equal interval optical comb teeth during transmission, and the dispersion delay is equal to the symbol duration of the time sequence electrical signal X, so as to ensure the interleaving in time and wavelength; finally, the product results of different wavelengths at the same time point are accumulated through the high-speed photodetector to obtain the convolution result Y of the time sequence electrical signal X and the weight vector W, and the photonic convolution result is obtained by using the computer and merged into an RGB three-channel image.
[0013] Further, the process of channel shuffling operation processing is as follows: reading the brightness information of the R, G and B channels of the color picture, then traversing each pixel point, randomly shuffling the information of the three channels corresponding to the pixel point, and after the traversal is completed, retaining the data of one channel as input data, and flattening the input data into a one-dimensional vector and inputting it into the AWG arbitrary signal generator to generate a time sequence electrical signal carrying multi-channel picture information.
[0014] Further, the power division function of the waveform shaper is used to realize space division multiplexing, that is, by adjusting the configuration file parameters of the waveform shaper, the optical frequency comb signal of the same wavelength is divided into multiple output ports, and then different attenuations are superimposed according to the set convolution kernel size and weight, and finally multiple photonic convolution kernels are efficiently constructed on multiple output ports.
[0015] Further, after the computer receives the convolution result output by the high-speed photodetector, a multi-feature dimension information integration technology is used to merge the convolution results of the plurality of convolution kernels into an RGB three-channel image.
[0016] Further, the multi-feature dimension information integration technology is: mapping convolution results of multiple convolution kernels into different channels, and finally integrating into a three-channel image containing three different feature maps.
[0017] In another aspect, the application also provides an application of the above-mentioned time, frequency and space three-dimensional interleaved photonic convolution accelerator for color images in a color image classification task.
[0018] Compared with the prior art, the application has the following beneficial effects:
[0019] 1. The time, frequency and space three-dimensional interleaved photonic convolution accelerator for color images introduced in the application introduces the channel shuffling idea into the photonic convolution accelerator system based on a separated photonic device, randomly shuffles the multi-channel data of a high-resolution color picture, then retains single-channel data as the input of photonic convolution, and performs convolution operation, so that the photonic convolution accelerator can extract cross-channel features under the limitation of only performing single-channel convolution, that is, the data containing multi-channel information is processed in a single channel, and multi-channel feature information is extracted by single convolution operation. Compared with the gray operation commonly used in color picture compression mode, the use of channel shuffling operation to process the training set has higher classification accuracy, which can further improve the accuracy of the entire system. Compared with the method of multiple convolution and multiple system construction, the method greatly reduces the workload and gives the photonic convolution accelerator the ability to process color pictures in a way that is easy to implement.
[0020] 2. The application utilizes the power division function of the waveform shaper to realize space division multiplexing, and under the condition of limited spectrum resources, multiple photonic convolution kernels can be efficiently constructed, the same frequency band optical comb signal can be repeatedly used in different spaces, the convolution efficiency of the photonic convolution accelerator is doubled, the working bandwidth requirement for the optical comb signal is reduced, the workload is greatly reduced, and the data amount of the output feature dimension is doubled. At the same time, the size and weight of the multiple photonic convolution kernels are arbitrarily adjustable, so that the photonic convolution accelerator can perform convolution in parallel and obtain convolution results of different feature dimensions.
[0021] 3. The photonic convolution accelerator based on the separated photonic device, proposes an optoelectronic hybrid convolutional neural network, superimposes different feature maps based on the original picture, integrates multiple feature information, fully utilizes the convolution results of the photonic convolution accelerator, so that the network can obtain data features from different feature dimensions, obtains a three-channel picture after multi-feature integration, and inputs the three-channel picture into the network for training and verification. Compared with the optoelectronic hybrid convolutional neural network trained by using only single feature information, the multi-feature information integration can obtain higher accuracy, and effectively breaks through the accuracy bottleneck in the case of limited feature information. BRIEF DESCRIPTION OF DRAWINGS
[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained based on these drawings.
[0023] Figure 1 The structure principle diagram of the photon convolution accelerator provided by the present application.
[0024] Figure 2 The optoelectronic hybrid convolutional neural network architecture provided by the present application.
[0025] Figure 3 The channel shuffling and multi-feature integration part simulation effect diagram provided by the present application.
[0026] Figure 4 The channel shuffling+multi-feature integration and original gray scale training set accuracy and validation set accuracy provided by the present application.
[0027] Figure 5 The channel shuffling+multi-feature integration and original gray scale training loss and validation loss provided by the present application.
[0028] Figure 6 The up big and down small non-negative convolution kernel optical frequency comb signal provided by the present application.
[0029] Figure 7 The all-1 convolution kernel optical frequency comb signal provided by the present application.
[0030] Figure 8 The positive and negative convolution effect provided by the present application.
[0031] Figure 9 The partial experimental results of constructing a plurality of convolution kernels provided by the present application.
[0032] Figure 10 The convolution effect corresponding to a plurality of convolution kernels provided by the present application.
[0033] Figure 11 The experimental prediction results provided by the present application.
[0034] Figure 12 The classification error picture provided by the present application. DETAILED DESCRIPTION
[0035] In order to better understand the technical solutions, the method of the present application will be described in detail below in combination with the drawings.
[0036] The present application proposes a time, frequency and space three-dimensional interlaced photon convolution accelerator for color images, which is constructed based on the interlaced time, wavelength and space dimensions, likeFigure 1 As shown, it is composed of optical frequency comb signal, power amplifier, waveform shaper, coupler, optical spectrum analyzer, polarization controller, polarization beam splitter, electro-optical modulator, dispersive fiber, photoelectric detector, AWG signal generator and oscilloscope. First, the high-resolution color picture is processed by channel shuffling operation, and the single-channel picture data is reserved, and then it is flattened into a one-dimensional vector and loaded onto the power of the time sequence electrical signal X, and its symbol rate is 1 / τ(baud), where τ is the symbol period. A plurality of convolution kernels are respectively expanded into one-dimensional vectors, and each one-dimensional vector corresponding to a convolution kernel is denoted by W. The power of the optical frequency comb signal is divided into different output ports through the power division function of the waveform shaper, and then the optical frequency comb signal is shaped according to the weight of the corresponding convolution kernel, so that the corresponding weights of the plurality of convolution kernels are respectively encoded into the power of the optical frequency comb signal of different output ports. Then, the electrical signal X modulates the optical frequency comb signal through the electro-optical modulator to generate the product result of the weight vector W. Then, through the dispersive fiber, an equal dispersion delay is superimposed on the equal interval optical comb teeth during transmission, and the dispersion delay is equal to the symbol duration of X, so as to ensure the interleaving in time and wavelength. Finally, the product results of different wavelengths at the same time point are accumulated through the high-speed photoelectric detector to obtain the convolution result Y of X and W, and the photonic convolution result is obtained by using a computer.
[0037] The channel shuffling operation first reads the brightness information of the R, G and B channels of the color picture, and then traverses each pixel point to randomly shuffle the information of the three channels corresponding to the point. After the traversal is completed, the data of one of the channels is reserved as the input data. The input data is flattened into a one-dimensional vector and input to the AWG arbitrary signal generator to generate a time sequence electrical signal carrying multi-channel picture information. In the case of limited frequency resources, space division multiplexing relies on the power division function of the waveform shaper, that is, by adjusting the configuration file parameters of the waveform shaper, a small amount of power loss is used to divide the optical frequency comb signal of the same wavelength into multiple output ports, and then different attenuations are superimposed according to the set convolution kernel size and weight, and finally multiple photonic convolution kernels are efficiently constructed on multiple output ports.
[0038] After receiving the signal output by the detector in the photonic convolution accelerator, the application introduces a multi-feature dimension information integration technology, which adopts a method of merging into an RGB three-channel image, that is, the convolution results of multiple convolution kernels are mapped into different channels, and finally integrated into a three-channel image containing three different feature maps, which is input into the subsequent electronic neural network for training and verification.
[0039] The optoelectronic hybrid convolutional neural network architecture based on the photonic convolution accelerator is as shown in Figure 2The first layer is a photonic convolution accelerator, responsible for channel shuffling of the picture and feature extraction of different convolution kernels; the second layer is a feature integration layer, completed by a computer, responsible for recovering different feature results extracted by the photonic convolution accelerator, and integrating and adjusting the picture size; the third layer is a planar convolution layer, containing three convolution kernels of different sizes, responsible for extracting feature information under different sizes of receptive fields; and the last layer is a MobileNet neural network, responsible for further processing the output data of the previous network to obtain the final classification result.
[0040] The present application is based on the realization of a photonic convolution accelerator based on a separate photonic device, combined with an electronic neural network, and can be applied to color image classification tasks.
[0041] In order to prove the effectiveness of the present application, first, the convolution results of channel shuffling, multi-feature integration processing and their combination are obtained by simulation method, and are input into the electronic neural network for training and verification, and the relevant data are recorded and analyzed. Subsequently, the space division multiplexing of the optical frequency comb is completed in the experiment, thereby constructing multiple photonic convolution kernels to obtain photonic convolution results of different feature dimensions, and then using the previously trained electronic neural network for verification.
[0042] The following verifies the practicability of the present application from two examples, which are the influence of channel shuffling and multi-feature integration operation on network accuracy, and the implementation of space division multiplexing and verification of experimental results.
[0043] 1. Influence of channel shuffling and multi-feature integration operation on network accuracy
[0044] In order to explore the influence of channel shuffling and multi-feature integration operation on network accuracy, this example uses a cat and dog dataset for system verification, and the training set of the dataset contains about 4000 cat and dog pictures each, and the verification set contains about 1000 cat and dog pictures each. First, the channel shuffling operation is completed on the training set by the computer, and then different photonic convolution results after shuffling are simulated, and then different features are integrated, input into the network for training, and the classification accuracy of the network is obtained. The simulation effect diagram of the channel shuffling and multi-feature integration part obtained by simulation is as shown in Figure 3 The classification accuracy of the network is shown in Table 1. The planar convolution layer in the electronic neural network uses convolution kernels of sizes 1x1, 3x3 and 9x9 respectively, the learning rate of the MobileNet neural network is 0.00001, the Adam optimizer is used, the loss function uses binary_crossentropy binary cross entropy, the activation function uses Sigmoid, and the training parameters are about 3230000, and the average result of 10 times training is taken. In order to make the model fit faster, the pre-training training method is adopted.
[0045] For channel shuffle operation, the training set is subjected to channel shuffle and grayscale operation. The classification accuracy results of channel shuffle + multi-feature integration are shown in Table 1.
[0046] Table 1 Classification accuracy of channel shuffle + multi-feature integration
[0047]
[0048] As can be seen from Table 1, as the receptive field becomes larger, the channel shuffle operation on the training set generally improves the network accuracy, and the grayscale operation only has higher accuracy when the size of the convolution kernel is 3x3. In the case of a 9x9 convolution kernel, the network accuracy after channel shuffling can reach 97.36%, which is 0.87% higher than the original grayscale operation under the same conditions. This is because the size of the convolution kernel is enlarged, which expands the range of the receptive field, and the cross-channel information in a larger area can be extracted accordingly, thereby improving the accuracy. For the case of a 3x3 convolution kernel, it may be due to the high resolution of the picture, and the 3x3 convolution kernel can extract more effective feature information than other sizes of convolution kernel, but it is still weaker than the network accuracy under the same conditions after channel shuffling.
[0049] For multi-feature integration operation, the data shows that, based on the original picture, superimposing its blurred feature map and sharpened feature map can generally improve the accuracy compared to not superimposing the feature map under various convolution kernel sizes. Among them, based on the grayscale image, the accuracy is improved most obviously under the condition of a 1x1 convolution kernel, and the average accuracy is improved by 1.49% after superimposing the blurred and sharpened feature maps; based on the channel shuffled picture, the accuracy is improved most obviously under the condition of a 1x1 convolution kernel, and the average accuracy is improved by 0.91% after superimposing the blurred and sharpened feature maps.
[0050] Through analysis, it is found that under certain conditions, both channel shuffling operation and multi-feature integration operation can improve the accuracy of the system. Channel shuffling has more obvious effect on improving the accuracy of the network under the condition of a large size of convolution kernel; multi-feature integration can significantly improve the accuracy of the network by superimposing different feature maps according to certain rules under the condition of a certain size of convolution kernel. The two functions can be combined in various ways under different conditions to further improve the accuracy of the network. Compared with the model accuracy of 96.24% of the original grayscale operation, the model accuracy is highest at 97.90% under the condition of a 3x3 convolution kernel after channel shuffling operation on the training set and superimposing the sharpened feature map, which is improved by 1.66%. The accuracy and loss of the two are shown in Figure 4 and Figure 5 It is worth mentioning that this case even surpasses the classification accuracy of color pictures, making the photonic convolution accelerator as the first accumulation layer of the optoelectronic hybrid convolutional neural network have a classification accuracy not inferior to the network with color pictures as the data set.
[0051] 2. Implementation of space division multiplexing and verification of experimental results
[0052] This example completes the construction of the photon convolution accelerator, and carries out photon convolution on ten pictures of cats and dogs. Twelve optical comb teeth near the 1550nm waveband are selected to construct the photon convolution kernel. The general steps of photon convolution are as follows: (1) picture data initialization; (2) automatic selection and positioning of optical frequency comb; (3) flattening of optical frequency comb; (4) shaping of optical frequency comb; (5) obtaining and restoring photon convolution results.
[0053] First, perform picture data preprocessing. First, use a computer to randomly shuffle the three-channel data of each pixel point of a high-resolution color picture. After traversing all pixel points, retain the complete data of one channel, then expand it into a one-dimensional vector, and input it into the AWG arbitrary signal generator in txt file format. Finally, generate a time sequence electrical signal carrying the multi-channel information of the picture.
[0054] Second, complete the automatic selection and positioning of the optical frequency comb. That is, according to the set system operating wavelength, locate the 12 optical comb teeth near the target wavelength that meet the set minimum signal-to-noise ratio, and eliminate the influence of interfering comb teeth through frequency shift correction to complete the precise positioning of the specified optical comb teeth.
[0055] Third, perform the flattening operation of the optical frequency comb. Through the waveform shaper, take the lowest power in the selected optical comb teeth as the target, and add attenuation to the optical comb teeth at other positions. Through cyclic correction, ensure that the power error of the final 12 optical comb teeth is within the allowable range.
[0056] Fourth, perform the shaping operation of the optical frequency comb, that is, according to the set convolution kernel size and weight, add attenuation to the flattened optical comb teeth. The photon convolution kernel realized in this process expresses the different weights of the convolution kernel through the power difference of different optical comb teeth. Take the power of the optical comb tooth corresponding to the maximum weight in the convolution kernel as the reference, add the attenuation corresponding to each weight, and the calculation formula of the attenuation is:
[0057] P i =10×log10(A max / A i )
[0058] Where i represents the serial number of the 12 weights, A max is the absolute value of the maximum weight, and A i is the absolute value of the i-th weight.
[0059] The last step is to obtain and restore the photon convolution results. Read the one-dimensional electrical signal of the high-speed photodetector through the oscilloscope, then use the computer to obtain the one-dimensional electrical signal, and then cut and splice according to a certain length to finally restore it to a two-dimensional picture.
[0060] The method for realizing positive-negative convolution in the present example is to subtract the convolution results of two non-negative convolution kernels, thereby obtaining the result of positive-negative convolution. Through space division multiplexing, a non-negative convolution kernel with high-low from top to bottom and an all-1 convolution kernel are respectively constructed at the 1 and 2 output ports of the waveform shaper, as shown in Figure 6 , Figure 7 respectively, and then the positive-negative convolution is completed through one convolution operation, as shown in Figure 8 .
[0061] In order to comprehensively test the convolution effect of the photon convolution accelerator, various convolution kernels are tried, and part of the experimental results are shown in Figure 9 , and the corresponding convolution effect of a high-resolution color picture through channel shuffling operation is shown in Figure 10 .
[0062] Using the ten pictures of cats and dogs obtained through photon convolution, the original convolution feature map and the blurred convolution feature map are integrated to verify the accuracy of the network model obtained through simulation in example 1, and the prediction result is shown in Figure 11 , and the accuracy reaches 95%. It is found that the model can accurately classify most cat and dog pictures. Only one picture has a prediction error, as shown in Figure 12 , and the reasons for this situation may be the instability of the experimental instrument, the existence of noise in the optical frequency comb signal and the existence of interference in the experimental environment. By replacing the experimental instrument with better performance, using an optical frequency comb signal with better signal-to-noise ratio, shielding the interference of the experimental environment and other methods, the influence of noise can be further weakened, a more stable experimental system can be obtained, and a more ideal convolution result can be obtained.
[0063] The above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features, but these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A time-frequency-spatial three-dimensional interwoven photonic convolution accelerator for color images, characterized in that, The system includes an optical frequency comb signal, a power amplifier, a waveform shaper, a coupler, a spectrometer, a polarization controller, a polarization beam splitter, an electro-optic modulator, a dispersive fiber, a high-speed photodetector, an AWG signal generator, and an oscilloscope, and the following processing steps: A high-resolution color image first undergoes channel shuffling to retain single-channel image data, which is then flattened into a one-dimensional vector and loaded onto the power of a time-series electrical signal X, with a symbol rate of 1 / τ, where τ is the symbol period. Then, multiple convolutional kernels are unfolded into one-dimensional vectors, each represented by a vector W. The power of the optical frequency comb signal is divided to different output ports using the power-splitting function of the waveform shaper, and then shaped according to the weights of the corresponding convolutional kernels, so that the corresponding weights of the multiple convolutional kernels are encoded onto the power of the optical frequency comb signal at different output ports. Finally, the time-series electrical signal X is modulated by the electro-optic modulator to produce a product with vector W. Then, through dispersive optical fiber, an equal dispersive delay is superimposed on the equally spaced optical comb teeth during transmission. The magnitude of this dispersive delay is equal to the symbol duration of the time-series electrical signal X, thus ensuring staggered time and wavelength. Finally, the product results of different wavelengths at the same time point are accumulated by a high-speed photodetector to obtain the convolution result Y of the time-series electrical signal X and vector W. Then, a computer is used to obtain the photon convolution result and merge it into an RGB three-channel image.
2. The time-frequency-spatial three-dimensional interwoven photonic convolution accelerator for color images according to claim 1, characterized in that, The channel shuffling operation process is as follows: read the brightness information of the R, G, and B channels of the color image, then traverse each pixel, randomly shuffle the information of the three channels corresponding to the pixel, retain the data of one channel as input data after traversal, flatten the input data into a one-dimensional vector and input it into the AWG arbitrary signal generator to generate a time-series electrical signal carrying multi-channel image information.
3. The time-frequency-spatial three-dimensional interwoven photonic convolution accelerator for color images according to claim 1, characterized in that, Space division multiplexing is achieved through the power division function of the waveform shaper. That is, by adjusting the configuration parameters of the waveform shaper, the optical frequency comb signal of the same wavelength is divided into multiple output ports. Then, according to the set convolution kernel size and weight, different attenuations are superimposed, and finally multiple photonic convolution kernels are efficiently constructed on multiple output ports.
4. The time-frequency-spatial three-dimensional interwoven photonic convolution accelerator for color images according to claim 1, characterized in that, After the computer receives the convolution result output by the high-speed photodetector, it uses multi-feature dimension information integration technology to merge the convolution results of multiple convolution kernels into an RGB three-channel image.
5. The time-frequency-spatial three-dimensional interwoven photonic convolution accelerator for color images according to claim 4, characterized in that, The multi-feature dimension information integration technology is to map the convolution results of multiple convolution kernels to different channels, and finally integrate them into a three-channel image containing three different feature maps.
6. The application of the time-frequency-spatial three-dimensional interleaved photonic convolution accelerator for color images according to any one of claims 1-5 in the task of color image classification.