Optoelectronic Fusion Convolutional Neural Network System Based on Temporal Talbot Effect

Abstract

The present invention discloses a convolutional neural network system of optoelectronic fusion based on time-domain Talbot effect, comprising: a light pulse generating device, configured to generate a first light pulse cluster and a second light pulse cluster; a convolution device, configured to generate a third light pulse cluster according to the first light pulse cluster, and perform convolution calculation on the third light pulse cluster based on the time-domain Talbot effect to generate a convolution calculation result, wherein the amplitude of the third light pulse cluster represents the data to be processed; and a fully connected device, configured to generate a fourth light pulse cluster according to the second light pulse cluster and the convolution calculation result, and perform recognition calculation on the fourth light pulse cluster based on the time-domain Talbot effect to generate a recognition calculation result.

Description

Technical Field

[0001] The present invention relates to the technical field of microwave photonics, and particularly to an optoelectronic fusion convolutional neural network system based on the time-domain Talbot effect. Background Art

[0002] Biological neural networks endow organisms with consciousness, help organisms think and act, and make organisms intelligent. Inspired by biological neural networks, humans have attempted to simulate and imitate biological neural networks and invented artificial neural networks. As an important type of artificial neural network, convolutional neural networks extract features from input signals through processes such as convolution and pooling, reducing the number of parameters to be optimized; secondly, by training to obtain the parameters for feature extraction, artificial feature extraction is avoided. These advantages have enabled convolutional neural networks to be widely applied in fields such as image classification and speech recognition.

[0003] However, with the progress of society and the development of technology, the amount of data that needs to be recognized and processed is increasing, and traditional convolutional neural networks based on the von Neumann architecture are increasingly facing bottlenecks such as processing speed, latency, and energy consumption. Summary of the Invention

[0004] In view of this, the main purpose of the present invention is to provide an optoelectronic fusion convolutional neural network system based on the time-domain Talbot effect, in order to at least partially solve at least one of the above-mentioned technical problems.

[0005] To achieve the above purpose, the technical solution of the present invention includes:

[0006] As an aspect of the present invention, there is provided an optoelectronic fusion convolutional neural network system based on the time-domain Talbot effect, including:

[0007] An optical pulse generating device configured to generate a first optical pulse cluster and a second optical pulse cluster;

[0008] A convolutional device configured to generate a third optical pulse cluster according to the first optical pulse cluster, and perform convolutional calculation on the third optical pulse cluster based on the time-domain Talbot effect to generate a convolutional calculation result, wherein the amplitude of the third optical pulse cluster represents the data to be processed; and

[0009] A fully connected device configured to generate a fourth optical pulse cluster according to the second optical pulse cluster and the convolutional calculation result, and perform recognition calculation on the fourth optical pulse cluster based on the time-domain Talbot effect to generate a recognition calculation result.

[0010] Based on the above technical solution, the present invention has at least one or a part of the following beneficial effects compared with the prior art:

[0011] By characterizing the data to be processed with the amplitude of a cluster of optical pulses, and performing convolution and recognition calculations in sequence using a convolution device and a fully connected device, a photoelectric fusion convolutional neural network based on the time-domain Talbot effect is simulated.

[0012] Since this system uses photon technology, it has the advantages of large bandwidth, low latency, low power consumption, and immunity to electromagnetic interference, thus improving the calculation speed of convolution calculation and recognition calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a schematic diagram of a photoelectric fusion convolutional neural network system based on the time-domain Talbot effect provided by an embodiment of the present invention;

[0014] Figure 2 is a schematic diagram of a convolution device provided by an embodiment of the present invention;

[0015] Figure 3 is a schematic diagram of a convolution calculation module provided by an embodiment of the present invention;

[0016] Figure 4 is a schematic diagram of the principle of generating the time-domain Talbot effect in a dispersive medium provided by an embodiment of the present invention;

[0017] Figure 5 is a schematic diagram of a first output module provided by an embodiment of the present invention;

[0018] Figure 6 is a schematic diagram of the data to be processed provided by an embodiment of the present invention;

[0019] Figure 7 is a schematic diagram of the principle of the time-domain Talbot effect provided by another embodiment of the present invention;

[0020] Figure 8 is a schematic diagram of the valid data provided by an embodiment of the present invention;

[0021] Figure 9 is a schematic diagram of a photoelectric conversion sub-module provided by an embodiment of the present invention;

[0022] Figure 10 is a schematic diagram of a fully connected device provided by an embodiment of the present invention;

[0023] Figure 11 is a schematic diagram of a recognition calculation module provided by an embodiment of the present invention;

[0024] Figure 12 is a schematic diagram of the principle of generating the time-domain Talbot effect in a dispersive medium provided by another embodiment of the present invention;

[0025] Figure 13 is a schematic diagram of a second output module provided by an embodiment of the present invention; and

[0026] Figure 14 It is a schematic diagram of the second output module provided by another embodiment of the present invention. Specific embodiments

[0027] The present invention provides an optoelectronic fusion convolutional neural network system based on the time-domain Talbot effect, including an optical pulse generating device, a convolutional device, and a fully connected device.

[0028] The optical pulse generating device is configured to generate a first optical pulse cluster and a second optical pulse cluster.

[0029] The convolutional device is configured to generate a third optical pulse cluster according to the first optical pulse cluster, and perform convolutional calculation on the third optical pulse cluster based on the time-domain Talbot effect to generate a convolutional calculation result, where the amplitude of the third optical pulse cluster represents the data to be processed. And

[0030] The fully connected device is configured to generate a fourth optical pulse cluster according to the second optical pulse cluster and the convolutional calculation result, and perform recognition calculation on the fourth optical pulse cluster based on the time-domain Talbot effect to generate a recognition calculation result.

[0031] By representing the data to be processed with the amplitude of the optical pulse cluster, and sequentially performing convolutional and recognition calculations using the convolutional device and the fully connected device, an optoelectronic fusion convolutional neural network based on the time-domain Talbot effect is constituted.

[0032] Since this system uses photon technology, it has the advantages of large bandwidth, low latency, low power consumption, and immunity to electromagnetic interference, thereby improving the calculation speed of convolutional calculation and recognition calculation.

[0033] The following will describe in detail the specific components and structures of the optoelectronic fusion convolutional neural network system of the present invention based on the time-domain Talbot effect with reference to the accompanying drawings.

[0034] In the following description, specific details are set forth in order to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the spirit of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0035] As Figure 1 shown, the present invention provides an optoelectronic fusion convolutional neural network system based on the time-domain Talbot effect, including an optical pulse generating device 1, a convolutional device 2, and a fully connected device 3.

[0036] The optical pulse generating device 1 is configured to generate a first optical pulse cluster and a second optical pulse cluster.

[0037] According to an embodiment of the present disclosure, the optical pulse generating device 1 may include a mode-locked laser and a beam splitter. The mode-locked laser is configured to emit an initial optical pulse cluster, and the beam splitter is configured to split the initial optical pulse cluster to split it into a first optical pulse cluster and a second optical pulse cluster.

[0038] According to an embodiment of the present invention, the optical pulses generated by the mode-locked laser have a comb-shaped spectrum.

[0039] According to an embodiment of the present invention, the mode-locked laser may be replaced by a combination of a continuous laser and an electro-optic modulator modulated by an electrical pulse signal or a directly modulated laser modulated by an electrical pulse signal.

[0040] The convolution device 2 is configured to generate a third optical pulse cluster according to the first optical pulse cluster and perform a convolution calculation on the third optical pulse cluster based on the time-domain Talbot effect to generate a convolution calculation result. The amplitude of the third optical pulse cluster represents the data to be processed.

[0041] The fully connected device 3 is configured to generate a fourth optical pulse cluster according to the second optical pulse cluster and the convolution calculation result and perform an identification calculation on the fourth optical pulse cluster based on the time-domain Talbot effect to generate an identification calculation result.

[0042] According to an embodiment of the present invention, with reference to Figure 2 , the convolution device 2 includes a first optical pulse modulation module 21, a convolution calculation module 22, and a first output module 23.

[0043] The first optical pulse modulation module 21 is configured to modulate the amplitude of the first optical pulse cluster and output an initial third optical pulse cluster. The optical pulses in the initial third optical pulse cluster have a comb-shaped spectrum, and the comb-shaped spectrum has a plurality of spectral components.

[0044] According to an embodiment of the present invention, the first optical pulse modulation module 21 may include an arbitrary waveform generator and a first electro-optic modulator. The arbitrary waveform generator is configured to generate a voltage waveform corresponding to the data to be processed, and the first electro-optic modulator is configured to load the voltage waveform onto the first optical pulse cluster so that the amplitude of the first optical pulse cluster represents the data to be processed, thereby generating an initial third optical pulse cluster.

[0045] According to an embodiment of the present invention, the arbitrary waveform generator may be replaced by a programmable pulse generator (PPG), a combination of any one or more of a field programmable gate array (FPGA), a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), and an analog-to-digital converter 332, but is not limited thereto. It may also be a combination of other units capable of performing logical operations and a digital-to-analog converter or other devices capable of generating arbitrary waveforms.

[0046] According to an embodiment of the present invention, the arbitrary waveform generator and the first electro-optic modulator can be connected by a cable.

[0047] The convolution calculation module 22 is configured to edit a plurality of spectral components of the initial third optical pulse cluster to obtain a third optical pulse cluster, where the third optical pulse cluster is loaded with a target convolution kernel; and perform convolution calculation on the third optical pulse cluster based on the time-domain Talbot effect, and output a fifth optical pulse cluster.

[0048] According to an embodiment of the present invention, the target convolution kernel can be obtained through computer simulation training.

[0049] According to an embodiment of the present invention, by using the convolution calculation module 22, the target convolution kernel can be loaded onto the spectral components of the third optical pulse cluster, and since the amplitude of the third optical pulse cluster represents the data to be processed, thus, based on the time-domain Talbot effect, the data to be processed can be optically convolved using the target convolution kernel.

[0050] The first output module 23 is configured to generate a first electrical signal according to the fifth optical pulse cluster, where the first electrical signal represents the convolution calculation result.

[0051] According to an embodiment of the present invention, with reference to Figure 3 , the convolution calculation module 22 includes a first dispersion medium 221 and a first waveform shaper 222.

[0052] The first dispersion medium 221 is configured to generate the time-domain Talbot effect to delay-align different spectral components of different optical pulses in the initial third optical pulse cluster, and generate a first optical pulse cluster to be processed.

[0053] According to an embodiment of the present invention, the first dispersion medium 221 can delay-align different spectral components in different optical pulses by generating the time-domain Talbot effect, and the process of delay alignment is the process of implementing convolution calculation.

[0054] According to an embodiment of the present invention, the mode-locked laser has a pulse repetition period.

[0055] According to an embodiment of the present invention, the dispersion coefficient, length of the first dispersion medium 221, and the pulse repetition period of the mode-locked laser satisfy the conditions of the time-domain Talbot effect:

[0056]

[0057] where s is any natural number, T is the pulse repetition period, is the dispersion coefficient, and L is the length of the dispersion medium.

[0058] The first waveform shaper 222 is configured to shape the spectral components of the first optical pulse cluster to be processed, so that the target convolution kernel is loaded onto the first optical pulse cluster to be processed, and beam splitting is performed according to the target convolution kernel represented by each spectral component in the first optical pulse cluster to be processed and the positive or negative value of the value in the target convolution kernel, and the fifth optical pulse cluster is output.

[0059] According to an embodiment of the present invention, the first waveform shaper 222 can first filter the first optical pulse cluster to be processed. Specifically, the first waveform shaper 222 can filter the first optical pulse cluster to be processed according to the number of values in the target convolution kernel and the number of convolution kernels. For example, if the target convolution kernel includes two 3x3 convolution kernels, thus, the first waveform shaper 222 can only allow 18 spectral components in the first optical pulse to pass through the first waveform shaper 222.

[0060] According to an embodiment of the present invention, the number of target convolution kernels can be one or more.

[0061] According to an embodiment of the present invention, the fifth optical pulse cluster includes a first spectral component of the fifth optical pulse cluster representing non - negative values in the target convolution kernel and a second spectral component of the fifth optical pulse cluster representing negative values in the convolution kernel.

[0062] The first waveform shaper 222 includes an output port; wherein, the number of output ports is determined by the number of the first spectral component and the second spectral component.

[0063] According to an embodiment of the present invention, the number of output ports of the first waveform shaper 222 can be twice the number of target convolution kernels.

[0064] According to an embodiment of the present invention, the principle of generating the time - domain Talbot effect in a dispersive medium is as Figure 4 shown. As an example, Figure 4 the comb - shaped spectrum of the initial optical pulse cluster generated by the mode - locked laser in [] includes four spectral components. However, it should be noted that in the actual application process, the number of spectral components can be determined according to the number of parameters in the convolution kernel and the number of convolution kernels. For example, when the target convolution kernel includes two 3x3 convolution kernels, the spectrum of the initial optical pulse cluster can include 18 spectral components.

[0065] As an example, the following takes the first optical pulse modulation module 21 generating an initial third optical pulse cluster containing only four spectral components as an example to introduce the time - delay alignment and calculation process of the optical pulse in the dispersive medium.

[0066] Figure 4 where X i1 to X i4 respectively represent the initial third optical pulse 1 to the initial third optical pulse 4 input into the dispersive medium. The amplitude of the initial third optical pulse cluster represents the data to be convolved, Xo1 to X o4 represent the first optical pulses to be processed 1 to 4 output from the dispersion medium, W1 to W 4 represent the four spectral components of the initial third optical pulse cluster.

[0067] After generating the time-domain Talbot effect in the dispersion medium, from the optical pulse X i1 ~X i4 The different spectral components are respectively delayed by 0 to 3 pulse repetition periods due to the dispersion effect. Then, when the first optical pulse cluster to be processed is output from the dispersion medium, X o1 's amplitude is the superposition of the different spectral components from the initial third optical pulse 1 to the initial third optical pulse 4, that is: X o1 = X il W 1 + X i2 W 2 + X i3 W 3 + X i4 W 4 .

[0068] The first optical pulse cluster output from the dispersion medium is input into the first waveform shaper 222. The first waveform shaper 222 first shapes the spectral components of the first optical pulse cluster to be processed, so that the target convolution kernel can be loaded onto the spectral components of the first optical pulse cluster to represent the target convolution kernel through optical information. For example, the target convolution kernel can be a 2x2 convolution kernel The first waveform shaper 222 shapes the four spectral components W 1 to W 4 of the first optical pulse cluster to be processed according to the absolute values of the parameters in the 2x2 convolution kernel. Thus, W 1 can represent the absolute value 2 of the parameter -2 in the first row and first column of the convolution kernel, W 2 can represent the absolute value 2 of the parameter -2 in the first row and second column of the convolution kernel, and so on. W 4 can represent the parameter 1 in the second row and second column of the convolution kernel.

[0069] In the above formula X o1 = X i1 W 1 + X i2 W 2 + X i3 W 3 + X i4 W 4 , for example, X i1 to X i4 are 1, 2, 3, and 4 respectively. After the first waveform shaper 222 shapes the spectral components of the first optical pulse cluster to be processed, W1 to W 4 represent 2, 2, 1, and 1 respectively. Therefore, the above formula can be expressed as X o1 = 1×2 + 2×2 + 3×1 + 4×1. Among them, the sign of the negative value in the target convolution kernel is achieved by subtracting the spectral components representing non - negative values and negative values in the balanced photodetector 331.

[0070] After the first waveform shaper 222 shapes the spectral components of the first optical pulse cluster to be processed, the first waveform shaper 222 can also determine from which port to output this spectral component according to the positive or negative value of the value in the convolution kernel represented by the spectral component and the convolution kernel to which the spectral component belongs. The spectral components from the same convolution kernel are output from two ports. Among them, the spectral components representing negative values in the convolution kernel are output from one port, and the spectral components representing non - negative values in the convolution kernel are output from the other port. Taking the target convolution kernel including the above 2x2 convolution kernel and the third optical pulse cluster including four spectral components W 1 to W 4 as an example, the values in the convolution kernel represented by the spectral components W 1 and W 2 are negative values, and the values in the convolution kernel represented by W 3 and W 4 are non - negative values. Then W 1 and W 2 are output from the first port, and W 3 and W 4 are output from the second port. If the target convolution kernel includes another 2x2 convolution kernel in addition to the above 2x2 convolution kernel, among the four spectral components corresponding to the other 2x2 convolution kernel, the spectral components representing negative values in the convolution kernel are output from the third port, and the spectral components representing non - negative values in the convolution kernel are output from the fourth port.

[0071] According to an embodiment of the present invention, the second spectral component may include W 1 and W 2 ; the first spectral component may include W 3 and W 4 .

[0072] According to an embodiment of the present invention, the first waveform shaper 222 can be used to shape spectral components and can also be used to split spectral components. Specifically, the first waveform shaper 222 can shape each spectral component of the comb spectrum, so as to load the target convolution kernel onto the spectrum; another function of the first waveform shaper 222 is to split different spectral components of the comb spectrum according to the target convolution kernel and the positive or negative value of the median value in the target convolution kernel. Among them, the number of output ports of the first waveform shaper 222 is twice the number of target convolution kernels, indicating that the spectral components representing a target convolution kernel are output from two ports of the first waveform shaper 222 according to the positive or negative value of the convolution kernel median value, with negative values output from one port and non-negative values output from one port.

[0073] According to an alternative embodiment of the present invention, the first waveform shaper 222 can also select a corresponding number of spectral components according to the size of a target convolution kernel (for example, two 3x3 convolution kernels select 9 spectral components), then divide these spectral components into several parts with equal power according to the number of target convolution kernels (for two 3x3 convolution kernels, divide them into two equal parts), and then load the target fully connected weights and output according to the positive or negative of the convolution kernel for the spectral components after power splitting.

[0074] It should be noted that in the convolution device provided by the embodiment of the present invention, the output end of the first dispersion medium is connected to the input end of the first waveform shaper. However, those skilled in the art can understand that it can also be the output end of the first waveform shaper connected to the input end of the first dispersion medium; that is to say, the order of using the first dispersion medium to generate the time-domain Talbot effect and using the first waveform shaper to edit the spectrum of the optical pulse can be reversed, which has no impact on the convolution calculation result.

[0075] According to an embodiment of the present invention, referring to Figure 5 , the first output module 23 includes an optical switch 231, a first optical coupler 232a, a second optical coupler 232b, and a photoelectric conversion sub-module 233.

[0076] The optical switch 231 is configured to filter the spectral components of the fifth optical pulse cluster according to a preset condition to generate an effective optical pulse cluster. Among them, the effective optical pulses in the effective optical pulse cluster include a first effective spectral component representing the non-negative values in the convolution kernel and a second effective spectral component representing the negative values in the convolution kernel; wherein, the number of optical switches 231 matches the number of target convolution kernels.

[0077] According to an embodiment of the present invention, the optical switch 231 filters the spectral components of the fifth optical pulse cluster, and the preset condition for filtering is determined according to the input mode of the data to be processed, the number and size of the convolution kernels included in the target convolution kernel.

[0078] The following uses a specific example to exemplarily illustrate the filtering of the spectral components of the fifth optical pulse cluster by the optical switch 231. Those skilled in the art can understand that the following example is only used to help those skilled in the art understand the present invention and should not make any improper limitations on the present invention.

[0079] For example, as Figure 6 shown, the data to be processed can be picture data. Figure 7 It is a schematic diagram of the principle of the time-domain Talbot effect in another embodiment of the present invention.

[0080] First, the picture data can be processed into one-dimensional data, that is, X 1 , X 2 , X 3 , X 4 , …, X 13 , X 14 , X 2 , X 15 , X 4 , X 16 , …, and then the processed one-dimensional data is input into an arbitrary waveform generator.

[0081] As Figure 7 shown, assuming the target convolution kernel is then the first optical pulse cluster to be processed obtained after generating the time-domain Talbot effect in the dispersive medium includes:

[0082] I 1 = X 1 W 4 ;

[0083] I 2 = X 1 W 3 + X 2 W 4 ;

[0084] I 3 = X 1 W 2 + X 2 W 3 + X 3 W 4 ;

[0085] I 4 = X 1 W 1 + X 2 W 2 + X 3 W 3 + X 4 W 4 ;

[0086] I 5 = X2 W 1 +X 3 W 2 +X 4 W 3 +X 5 W 4 ;

[0087] I 6 = X 3 W 1 +X 4 W 2 +X 5 W 3 +X 6 W 4 ;

[0088] I 7 = X 4 W 1 +X 5 W 2 +X 6 W 3 +X 7 W 4 ;

[0089] …

[0090] When the target convolution kernel includes a 2x2 convolution kernel, the obtained effective optical pulse clusters include I 4 , I 6 , …, when the target convolution kernel includes two 2x2 convolution kernels, the effective optical pulse clusters obtained during the parallel calculation of the two target convolution kernels respectively include the I of the two convolution kernels 4 , I 6 , …, by adding a delay of one pulse repetition period to the light at the two output ports representing one of the convolution kernels, the effective data of the two convolution kernels are cross-aligned, as Figure 8 shown; then the optical switch 231 is used to select the effective optical pulses respectively at a switching speed half of the optical pulse repetition frequency, and the output optical pulses are the convolution results of the two cross-input convolution kernels. In this case, the switching speeds of all the optical switches 231 are the same, and the two optical switches 231 on the negative and non-negative value channels representing the same convolution kernel are turned on and off simultaneously; the switching speeds of the optical switches 231 representing different convolution kernels are the same, but the switching directions are opposite (i.e., when the two optical switches 231 representing the first convolution kernel are in the on state, the two optical switches 231 representing the other convolution kernel are in the off state).

[0091] When the dimension and number of the convolution kernel, the data input method, the type of the input data, etc. change, the position of the effective data will change accordingly, and the switching speed of the optical switch 231 will also change accordingly.

[0092] So far, the filtering of the spectral components of the fifth optical pulse cluster by the optical switch 231 has been described by way of example in combination with the above examples.

[0093] The first optical coupler 232a is configured to receive and combine a plurality of first effective spectral components.

[0094] The second optical coupler 232b is configured to receive and combine a plurality of second effective spectral components.

[0095] The photoelectric conversion sub-module 233 is configured to generate a first electrical signal according to the optical pulses output by the first optical coupler 232a and the optical pulses output by the second optical coupler 232b.

[0096] Figure 9 is a schematic diagram of the photoelectric conversion sub-module according to an embodiment of the present invention.

[0097] According to an embodiment of the present invention, with reference to Figure 9 , the photoelectric conversion sub-module 233 includes a first balanced photodetector 331, a first analog-to-digital converter 332, a first logic operation unit 333, and a digital-to-analog converter 334.

[0098] According to an embodiment of the present invention, the first balanced photodetector 331 includes two input ports, which are respectively configured to receive the first effective spectral component and the second effective spectral component output by the first optical coupler 232a and the second optical coupler 232b. Then, the first balanced photodetector 331 subtracts the first effective spectral component and the second effective spectral component, and converts the subtraction result into an initial electrical signal.

[0099] According to an embodiment of the present invention, the first analog-to-digital converter 332 is configured to convert the initial electrical signal into an initial digital signal.

[0100] According to an embodiment of the present invention, the first logic operation unit 333 is configured to perform operations such as filtering and pooling on the initial digital signal to generate a second digital signal.

[0101] According to an embodiment of the present invention, the digital-to-analog converter is configured to convert the second digital signal into a first electrical signal.

[0102] According to an embodiment of the present invention, the convolution device 2 further includes an electrical amplifier.

[0103] According to an embodiment of the present invention, an electrical amplifier can be connected between the first electro-optic modulator and the arbitrary waveform generator to increase the driving voltage of the electro-optic modulator.

[0104] According to an embodiment of the present invention, the convolution device 2 further includes an arbitrary number of optical amplifiers.

[0105] According to an embodiment of the present invention, an optical amplifier may be connected between the first dispersion medium 221 and the first waveform shaper 222, but is not limited thereto. The optical amplifier may be connected to the output end of any device that can output optical pulses to increase the optical power.

[0106] According to an embodiment of the present invention, the convolutional device 2 further includes an arbitrary number of optical filters.

[0107] According to an embodiment of the present invention, an optical filter may be connected to the output end of the first electro-optic modulator, but is not limited thereto. The optical filter may be connected to the output end of any device that can output optical pulses to filter out unwanted spectral components.

[0108] According to an embodiment of the present invention, the convolutional device 2 further includes a radio frequency biaser.

[0109] According to an embodiment of the present invention, a radio frequency biaser may be connected between any waveform generator and the first electro-optic modulator to change the range of the voltage for driving the electro-optic modulator.

[0110] According to an embodiment of the present invention, referring to Figure 10 , the fully connected device 3 includes a second optical pulse modulation module 31, an identification calculation module 32, and a second output module 33.

[0111] The second optical pulse modulation module 31 is configured to modulate the amplitude of the second optical pulse cluster according to the first electrical signal and output an initial fourth optical pulse cluster, where the amplitude of the initial fourth optical pulse cluster represents the result of the convolutional calculation.

[0112] According to an embodiment of the present invention, the second optical pulse modulation module 31 may include a second electro-optic modulator, which is configured to load the voltage waveform of the first electrical signal onto the second optical pulse cluster, so that the amplitude of the second optical pulse cluster represents the convolutional calculation result data output by the convolutional device 2, thereby generating an initial fourth optical pulse cluster.

[0113] The identification calculation module is configured to edit multiple spectral components of the initial fourth optical pulse cluster to obtain a fourth optical pulse cluster, where the fourth optical pulse cluster is loaded with the target fully connected weight; and perform identification calculation on the fourth optical pulse cluster based on the time-domain Talbot effect and output a sixth optical pulse cluster.

[0114] According to an embodiment of the present invention, the target fully connected weight may be obtained through computer simulation training.

[0115] The second output module is configured to output the identification calculation result based on the sixth optical pulse cluster.

[0116] According to an embodiment of the present invention, referring to Figure 11, the recognition and calculation module includes a second dispersion medium 321 and a second waveform shaper 322.

[0117] The second dispersion medium 321 is configured to generate a time-domain Talbot effect to delay and align different spectral components of different optical pulses in the initial fourth optical pulse cluster, generating a second optical pulse cluster to be processed.

[0118] According to an embodiment of the present invention, the mode-locked laser has a pulse repetition period.

[0119] According to an embodiment of the present invention, the dispersion coefficient, length of the second dispersion medium 321, and the pulse repetition period of the mode-locked laser satisfy the conditions of the time-domain Talbot effect:

[0120]

[0121] where s is any natural number, T is the pulse repetition period, is the dispersion coefficient, and L is the length of the dispersion medium.

[0122] According to an embodiment of the present invention, the first dispersion medium 221 and the second dispersion medium 321 have the same dispersion amount, that is, the product of the dispersion coefficient and the length of the first dispersion medium 221 and the second dispersion medium 321 is equal.

[0123] The second waveform shaper 322 is configured to shape the spectral components of the second optical pulse cluster to be processed, so that the target fully connected weights are loaded onto the second optical pulse cluster to be processed, generating a fourth optical pulse cluster loaded with the target fully connected weights, and splitting each spectral component in the fourth optical pulse cluster according to a preset decision rule and the positive and negative of the weight value of the target fully connected weights, outputting a sixth optical pulse cluster.

[0124] According to an embodiment of the present invention, the preset decision rule is related to the processing result of the convolutional neural network system for the data to be processed. For example, when the convolutional neural network system recognizes numbers, a decision needs to be defined for each of the recognition results of the convolutional neural network system being the ten numbers from 0 to 9; when the convolutional neural network system recognizes cats and dogs in pictures, a decision can be defined for the recognition result of the convolutional neural network system being a cat or a dog respectively.

[0125] According to an embodiment of the present invention, the number of output ports of the second waveform shaper 322 can be twice the number of decisions.

[0126] According to an embodiment of the present invention, the sixth optical pulse cluster includes a third spectral component of the sixth optical pulse cluster representing non-negative target fully connected weights and a fourth spectral component of the sixth optical pulse cluster representing negative target fully connected weights.

[0127] The second waveform shaper 322 includes an output port; wherein, the number of output ports is determined by the number of the third spectral component and the fourth spectral component.

[0128] According to an embodiment of the present invention, the principle of generating the time-domain Talbot effect in the second dispersion medium 321 is as Figure 12 shown. As an example, Figure 11 the spectrum of the initial optical pulse train generated by the mode-locked laser in [] includes four spectral components. However, it should be noted that in the actual application process, the number of spectral components can be determined according to the number of target fully connected weights.

[0129] As an example, the following takes the second optical pulse modulation module 31 generating an initial fourth optical pulse train only containing four spectral components as an example to introduce the delay alignment and calculation process of the optical pulse in the dispersion medium.

[0130] Figure 12 X in [] i1 to X i4 respectively represent the initial fourth optical pulse 1 to the initial fourth optical pulse 4 input to the second dispersion medium 321. The amplitude of the initial fourth optical pulse train represents the convolution calculation result data calculated and output by the convolution device 2. X o1 to X o4 represent the second optical pulse to be processed 1 to the second optical pulse to be processed 4 output from the second dispersion medium 321. W 1 to W 4 represent the four spectral components of the initial fourth optical pulse train.

[0131] After generating the time-domain Talbot effect in the second dispersion medium 321, different spectral components from the optical pulses X i1 ~X i4 are respectively delayed by 0 to 3 pulse repetition periods due to the dispersion effect. Then, when the second optical pulse train to be processed is output from the dispersion medium, the amplitude of X o1 is the superposition of different spectral components from the initial fourth optical pulse 1 to the initial fourth optical pulse 4, that is: X o1 =X i1 W 1 +X i2 W 2 +X i3 W 3 +X i4 W 4 .

[0132] The second optical pulse cluster output by the second dispersion medium 321 is input into the second waveform shaper 322. The second waveform shaper 322 first shapes the spectral components of the second optical pulse cluster to be processed, so that the target fully connected weights can be loaded onto the spectral components of the second optical pulse cluster to be processed, and the target fully connected weights are represented by optical information. For example, the target fully connected weights may include four fully connected weight factors -2, -3, 4, and 1. The second waveform shaper 322 shapes the four spectral components W 1 to W 4 of the second optical pulse cluster to be processed according to the absolute values of the parameters in the target fully connected weights. Thus, W 1 can represent the absolute value 2 of the fully connected weight factor -2, W 2 can represent the absolute value 3 of the fully connected weight factor -3, and so on. W 4 can represent the fully connected weight factor 1.

[0133] In the above formula X o1 = X i1 W 1 + X i2 W 2 + X i3 W 3 + X i4 W 4 , for example, X i1 to X i4 are 1, 2, 3, and 4 respectively. After the second waveform shaper 322 shapes the spectral components of the second optical pulse cluster to be processed, W 1 to W 4 represent 2, 3, 4, and 1 respectively. Therefore, the above formula can be expressed as X o1 = 1×2 + 2×3 + 3×4 + 4×1. Among them, the signs of the negative values in the target fully connected weights are realized by subtracting the spectral components representing non-negative values and negative values in the balanced photodetector 331.

[0134] After the second waveform shaper 322 shapes the spectral components of the second optical pulse cluster to be processed, the second waveform shaper 322 can determine from which port this spectral component outputs according to the positive or negative of the target fully connected weight represented by the spectral component and the decision to which the spectral component belongs. The spectral components from the same decision are output from two ports. Among them, the spectral components representing negative target fully connected weights in the same decision are output from one port, and the spectral components representing non-negative target fully connected weights in the same decision are output from the other port.

[0135] According to an embodiment of the present invention, the second waveform shaper 322 can be used to shape spectral components and can also be used to split spectral components. Specifically, the second waveform shaper 322 can shape each spectral component of the comb spectrum, so as to load the target fully connected weight kernel onto the spectrum; another function of the second waveform shaper 322 is to split different spectral components of the comb spectrum according to the decision and the positive or negative of the target fully connected weight value in the same decision. Among them, the number of output ports of the second waveform shaper 322 is twice the number of decisions.

[0136] According to an alternative embodiment of the present invention, the second waveform shaper 322 can also select a corresponding number of spectral components according to the number of fully connected weight factors required for one decision, then divide these spectral components into several equal-power parts according to the number of decisions, and then load the target full connection weights onto the power-divided spectral components and output according to the positive or negative of the weight factors.

[0137] It should be noted that in the fully connected device provided by the embodiment of the present invention, the output end of the second dispersion medium is connected to the input end of the second waveform shaper. However, those skilled in the art can understand that it can also be the output end of the second waveform shaper connected to the input end of the second dispersion medium; that is to say, the order of using the second dispersion medium to generate the time-domain Talbot effect and using the second waveform shaper to edit the spectrum of the optical pulse can be reversed, which has no impact on the recognition calculation result.

[0138] According to an embodiment of the present invention, referring to Figure 13 , the second output module includes a balanced photodetector 331.

[0139] Each balanced photodetector 331 includes a first input port and a second input port. Among them, the first input port is configured to receive the third spectral component, and the second input port is configured to receive the fourth spectral component; or

[0140] The first input port is configured to receive the spectral component of the sixth optical pulse representing the non-positive value in the convolution kernel, and the second input port is configured to receive the spectral component of the sixth optical pulse representing the positive value in the convolution kernel.

[0141] The balanced photodetector 331 is configured to subtract the third spectral component from the fourth spectral component and convert the subtraction result into a second electrical signal.

[0142] According to an embodiment of the present invention, the balanced photodetector 331 can be replaced by a combination of two photodetectors with the same characteristics and a differentiator. The balanced photodetector 331 can also be replaced by a combination of two photodetectors with the same characteristics, an analog-to-digital converter, and a logic operation unit.

[0143] According to an embodiment of the present invention, referring toFigure 14 The second output module further includes an analog-to-digital converter 332 and a logic operation unit 333.

[0144] The analog-to-digital converter 332 is configured to convert the second electrical signals output by the plurality of balanced photodetectors 331 into digital signals respectively;

[0145] The logic operation unit 333 is configured to determine the digital signal with the largest signal value from the plurality of digital signals and use it as the recognition calculation result.

[0146] According to an embodiment of the present invention, the fully connected device 3 further includes any number of optical delay lines.

[0147] According to an embodiment of the present disclosure, an optical delay line can be connected between the second waveform shaper 322 and the balanced photodetector 331 to achieve alignment between lights of different channels.

[0148] According to an embodiment of the present invention, the fully connected device 3 further includes any number of optical amplifiers.

[0149] According to an embodiment of the present invention, an optical amplifier can be connected between the second dispersion medium 321 and the second waveform shaper 322, but not limited thereto. The optical amplifier can be connected to the output end of any device that can output optical pulses to increase the optical power.

[0150] According to an embodiment of the present invention, the fully connected device 3 further includes any number of optical filters.

[0151] According to an embodiment of the present invention, an optical filter can be connected to the output end of the second electro-optic modulator, but not limited thereto. The optical filter can be connected to the output end of any device that can output optical pulses to filter out unnecessary spectral components.

[0152] The optoelectronic fusion convolutional neural network system based on the time-domain Talbot effect provided by the embodiment of the present invention has scalability, that is, multiple convolutional devices 2 and multiple fully connected devices 3 can be cascaded to implement a more complex convolutional neural network.

[0153] The above specific embodiments further elaborate on the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An optoelectronic fusion convolutional neural network system based on the time-domain Talbot effect, comprising: an optical pulse generating device configured to generate a first optical pulse cluster and a second optical pulse cluster; a convolutional device configured to generate a third optical pulse cluster according to the first optical pulse cluster, and perform a convolutional calculation on the third optical pulse cluster based on the time-domain Talbot effect to generate a convolutional calculation result, wherein the amplitude of the third optical pulse cluster represents the data to be processed; and a fully connected device configured to generate a fourth optical pulse cluster according to the second optical pulse cluster and the convolutional calculation result, and perform an identification calculation on the fourth optical pulse cluster based on the time-domain Talbot effect to generate an identification calculation result; The convolutional device includes: a first optical pulse modulation module, a convolutional calculation module, and a first output module; the first optical pulse modulation module is configured to modulate the amplitude of the first optical pulse cluster and output an initial third optical pulse cluster, wherein the optical pulses in the initial third optical pulse cluster have a comb-shaped spectrum with multiple spectral components; the convolutional calculation module is configured to edit the multiple spectral components of the initial third optical pulse cluster to obtain the third optical pulse cluster, wherein the third optical pulse cluster is loaded with a target convolutional kernel; and perform a convolutional calculation on the third optical pulse cluster based on the time-domain Talbot effect to output a fifth optical pulse cluster; the first output module is configured to generate a first electrical signal according to the fifth optical pulse cluster, wherein the first electrical signal represents the convolutional calculation result; The fully connected device includes: a second optical pulse modulation module, an identification calculation module, and a second output module; the second optical pulse modulation module is configured to modulate the amplitude of the second optical pulse cluster according to the first electrical signal and output an initial fourth optical pulse cluster, wherein the amplitude of the initial fourth optical pulse cluster represents the convolutional calculation result; the identification calculation module is configured to edit the multiple spectral components of the initial fourth optical pulse cluster to obtain the fourth optical pulse cluster, wherein the fourth optical pulse cluster is loaded with a target fully connected weight; and perform an identification calculation on the fourth optical pulse cluster based on the time-domain Talbot effect to output a sixth optical pulse cluster; the second output module is configured to output the identification calculation result based on the sixth optical pulse cluster.

2. The convolutional neural network system according to claim 1, wherein, the convolutional calculation module includes: a first dispersive medium configured to generate the time-domain Talbot effect to delay-align different spectral components of different optical pulses in the initial third optical pulse cluster to generate a first optical pulse cluster to be processed; a first waveform shaper configured to shape the spectral components of the first optical pulse cluster to be processed so that the target convolutional kernel is loaded onto the first optical pulse cluster to be processed to generate the third optical pulse cluster, and split the beam according to the target convolutional kernel represented by each spectral component in the third optical pulse cluster and the positive and negative values of the values in the target convolutional kernel to output the fifth optical pulse cluster.

3. The convolutional neural network system according to claim 2, wherein, The fifth optical pulse cluster includes a first spectral component of the fifth optical pulse cluster representing non - negative values in the target convolution kernel and a second spectral component of the fifth optical pulse cluster representing negative values in the convolution kernel; The first waveform shaper includes an output port; wherein, The number of the output ports is determined by the number of the first spectral component and the second spectral component.

4. The convolutional neural network system according to claim 3, wherein, The first output module includes: An optical switch configured to filter the spectral components of the fifth optical pulse cluster according to a preset condition to generate an effective optical pulse cluster, wherein the effective optical pulses in the effective optical pulse cluster include a first effective spectral component of the effective optical pulses representing non - negative values in the convolution kernel and a second effective spectral component of the effective optical pulses representing negative values in the convolution kernel; wherein the number of the optical switches matches the number of the target convolution kernels; A first optical coupler configured to receive and combine a plurality of the first effective spectral components; A second optical coupler configured to receive and combine a plurality of the second effective spectral components; A photoelectric conversion sub - module configured to generate the first electrical signal according to the optical pulses output by the first optical coupler and the optical pulses output by the second optical coupler.

5. The convolutional neural network system according to claim 1, wherein, The recognition and calculation module includes: A second dispersive medium configured to generate the time - domain Talbot effect to delay and align different spectral components of different optical pulses in the initial fourth optical pulse cluster to generate a second optical pulse cluster to be processed; A second waveform shaper configured to shape the spectral components of the second optical pulse cluster to be processed so that the target fully - connected weights are loaded onto the second optical pulse cluster to generate the fourth optical pulse cluster loaded with the target fully - connected weights, and split the spectral components in the fourth optical pulse cluster according to a preset decision rule and the positive and negative values of the weights of the target fully - connected weights, and output the sixth optical pulse cluster.

6. The convolutional neural network system according to claim 5, wherein, The sixth optical pulse cluster includes a third spectral component of the sixth optical pulse cluster representing non - negative target fully - connected weights and a fourth spectral component of the sixth optical pulse cluster representing negative target fully - connected weights; The second waveform shaper includes an output port; wherein, The number of the output ports is determined by the number of the third spectral component and the fourth spectral component.

7. The convolutional neural network system according to claim 6, wherein, The second output module includes a balanced photodetector, wherein, Each balanced photodetector includes a first input port and a second input port, wherein the first input port is configured to receive the third spectral component, and the second input port is configured to receive the fourth spectral component; or The first input port is configured to receive the spectral components of the sixth optical pulse representing non - positive values in the convolution kernel, and the second input port is configured to receive the spectral components of the sixth optical pulse representing positive values in the convolution kernel; The balanced photodetector is configured to subtract the third spectral component from the fourth spectral component and convert the subtraction result into a second electrical signal.

8. The convolutional neural network system according to claim 7, wherein the second output module further comprises: an analog-to-digital converter configured to convert the second electrical signals output by the plurality of balanced photodetectors into digital signals respectively; a logic operation unit configured to determine the digital signal with the largest signal value from the plurality of digital signals and use it as the recognition calculation result.

Images