Photonic neural network

Abstract

The photon neural network comprises a first photon matrix calculation unit which is at least used for carrying out matrix multiplication operation on optical signals of a first array to obtain optical signals of a second array; the second photon matrix calculation unit is at least used for carrying out matrix multiplication operation on the optical signals of the third array to obtain optical signals of a fourth array; the dot product calculation unit is respectively coupled with the first photon matrix calculation unit and the second photon matrix calculation unit, and the dot product calculation unit is used for performing dot product calculation on the optical signal of the second array and the optical signal of the fourth array to obtain an optical signal of a fifth array; and the light receiving unit is coupled with the dot product calculation unit so as to receive the optical signal of the fifth array, and the light receiving unit acquires the processed signal based on the optical signal of the fifth array. By means of the scheme, vector multiplication and dot product operation can be achieved in the full optical domain, and when Transform calculation is carried out, the calculation efficiency is high, and power consumption is low.

Description

Technical Field

[0001] The present invention relates to the technical field of photonic artificial intelligence chips, and in particular to a photonic neural network. Background Art

[0002] Traditional artificial intelligence is mainly achieved through computing, that is, machine intelligence is achieved through programming and other means, especially deep learning. As a widely used technology, it constructs a feedforward neural network by adopting a multi-layer weight summation plus nonlinear activation method to achieve intelligent recognition, target detection, data analysis and other functions. This technology involves a large number of linear multiplication and addition operations, which consumes a lot of computing resources and brings great performance and power consumption challenges to current electronic computing chips. To this end, some studies in recent years have tried to complete these linear multiplication and addition operations in the optical domain, and use the advantages of high bandwidth, low latency and low energy consumption of light to accelerate the calculation of deep feedforward neural network models, including spatial modulators, Mach-Zehnder interferometers (MZI), semiconductor optical amplifiers (SOA), micro-ring resonators (MRR), etc.

[0003] At present, silicon-based optoelectronic hybrid artificial intelligence (AI) acceleration chips are mainly used for convolutional neural network computing acceleration. Today's mainstream artificial intelligence algorithms are large models (i.e., large language models) built on the Transformer architecture. Such large models have significant differences in computing characteristics from traditional convolutional neural networks. Specifically, convolutional neural networks implement feature extraction based on sliding convolution, while Transformer networks capture global dependencies based on the self-attention mechanism. This results in relatively low computing efficiency when using the traditional optical computing neural network architecture for convolutional calculations for Transformer calculations. Summary of the invention

[0004] The technical problem solved by the present invention is to provide an improved photonic neural network.

[0005] To solve the above technical problems, an embodiment of the present invention provides a photonic neural network, including: a first photon matrix calculation unit, used to perform matrix calculation on the optical signal of the first array to obtain the optical signal of the second array, wherein the matrix calculation at least includes matrix multiplication operation, and the optical signal of the first array is modulated according to the signal to be processed; a second photon matrix calculation unit, used to perform matrix calculation on the optical signal of the third array to obtain the optical signal of the fourth array, wherein the matrix calculation at least includes matrix multiplication operation, and the optical signal of the fourth array is modulated according to the signal to be processed; a dot product calculation unit, respectively coupled to the first photon matrix calculation unit and the second photon matrix calculation unit, the dot product calculation unit is used to perform dot product calculation on the optical signal of the second array and the optical signal of the fourth array to obtain the optical signal of the fifth array; a light receiving unit, coupled to the dot product calculation unit to receive the optical signal of the fifth array, and the light receiving unit obtains the processed signal based on the optical signal of the fifth array.

[0006] Optionally, the dot product calculation unit includes: n columns of first waveguides, used to receive the optical signal of the second array; n rows of second waveguides, used to receive the optical signal of the fourth array; and a plurality of coherent detection units arranged in an array, respectively coupled to the n columns of first waveguides and the n rows of second waveguides, so as to perform dot product calculation on the optical signal of the second array and the optical signal of the fourth array to obtain the optical signal of the fifth array.

[0007] Optionally, the n columns of first waveguides and the n rows of second waveguides have multiple intersections, and the multiple intersections correspond one-to-one to the multiple coherent detection units. For each of the coherent detection units, the first input end of the coherent detection unit is coupled to the first waveguide of the corresponding intersection, and the second input end of the coherent detection unit is coupled to the second waveguide of the corresponding intersection.

[0008] Optionally, the first waveguide and the second waveguide are not optically coupled at the intersection.

[0009] Optionally, for each of the coherent detection units, the coherent detection unit includes: a 2×2 optical amplitude modulation unit and a balanced detector, the two input ends of the 2×2 optical amplitude modulation unit are respectively coupled to the first waveguide and the second waveguide, the two output ends of the 2×2 optical amplitude modulation unit are respectively coupled to the balanced detector, and the output end of the balanced detector is coupled to the optical receiving unit.

[0010] Optionally, the first waveguide and the second waveguide are located in the same waveguide layer, and the first waveguide and the coherent detection unit are located in different waveguide layers.

[0011] Optionally, the dot product calculation unit also includes: a plurality of vertical coupling structures, corresponding one-to-one to the plurality of coherent detection units, and for each of the vertical coupling structures, the vertical coupling structure is used to couple the optical signal transmitted by the first waveguide and / or the second waveguide to the waveguide layer where the corresponding coherent detection unit is located.

[0012] Optionally, for any one of the first photon matrix calculation unit and the second photon matrix calculation unit, the photon matrix calculation unit includes: an optical modulation unit, used to modulate the signal to be processed into light; a plurality of optical amplitude modulation units arranged in an array, used to receive an optical signal of an input array and generate an optical signal of an output array, the optical signal of the input array is the optical signal of the first array or the optical signal of the third array, and the optical signal of the output array is the optical signal of the second array or the optical signal of the fourth array.

[0013] Optionally, the photon matrix calculation unit also includes m rows of input waveguides and n columns of output waveguides, m and n are both positive integers, the m rows of input waveguides and n columns of output waveguides have multiple intersections, the multiple optical amplitude modulation units correspond one-to-one to the multiple intersections, and for each of the optical amplitude modulation units, the input end of the optical amplitude modulation unit is coupled to the input waveguide of the corresponding intersection, and the output end of the optical amplitude modulation unit is coupled to the output waveguide of the corresponding intersection.

[0014] Optionally, the input waveguide and the output waveguide are made of different materials; and / or, the input waveguide and the output waveguide are located in different waveguide layers; and / or, the input waveguide and the output waveguide are not optically coupled at the intersection.

[0015] Optionally, the calculation results of multiple optical amplitude modulation units corresponding to the same column of output waveguides are output from the output waveguide in a time-sharing manner, or the calculation results of multiple optical amplitude modulation units corresponding to the same column of output waveguides are accumulated and output from the output waveguide.

[0016] Optionally, the photon matrix calculation unit further includes: a plurality of phase shifters arranged in an array, corresponding one-to-one to the plurality of optical amplitude modulation units, and for each phase shifter, the phase shifter is coupled to an output end of the corresponding optical amplitude modulation unit.

[0017] Optionally, for at least one of the matrix calculation and the dot product calculation, the calculation of the optical signal and the transmission of the calculation result of the optical signal are performed in different waveguide layers.

[0018] Optionally, the optical signal is calculated in a silicon waveguide layer, and the optical signal is transmitted in a silicon nitride waveguide layer.

[0019] Optionally, the light receiving unit is also coupled to the first photon matrix calculation unit to receive the optical signal of the second array, and the light receiving unit obtains a processed signal based on the optical signal of the second array; and / or, the light receiving unit is also coupled to the second photon matrix calculation unit to receive the optical signal of the fourth array, and the light receiving unit obtains a processed signal based on the optical signal of the fourth array.

[0020] Optionally, the photonic neural network is used at least for generative artificial intelligence, and the generative artificial intelligence is at least selected from: speech generation, image generation, and text generation.

[0021] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:

[0022] Compared with the existing photonic neural network for convolution calculation, which can only perform matrix multiplication operations, this embodiment can realize vector multiplication and dot product operations in the entire optical domain, so that the calculation efficiency and power consumption of the photonic neural network using this embodiment for Transformer calculation are higher. Specifically, the first photon matrix calculation unit performs a convolution operation on the processed signal to obtain a first convolution operation result (based on the optical signal carried by the second array), and the optical signal of the second array performs a convolution operation on the processed signal to obtain a second convolution operation result (based on the optical signal carried by the fourth array). The first convolution operation result and the second convolution operation result are respectively used as the rows and columns of the matrix to input the dot product calculation unit for dot product calculation, and the operation result that includes both feature extraction and dependency between features is obtained (based on the optical signal carried by the fifth array). In this way, an optical computing network architecture that simultaneously accelerates matrix multiplication calculations (for encoding) and dot product calculations (for obtaining dependencies) in the Transformer neural network is realized.

[0023] Furthermore, this implementation scheme is driven by the Transformer neural network algorithm architecture. By constructing two matrix networks (i.e., the first photon matrix calculation unit and the second photon matrix calculation unit), the input signals (i.e., the optical signals of the first array and the optical signals of the third array) are respectively encoded to obtain the corresponding query (query, q), key (key, k) and information extraction value (value, v). Afterwards, the output results of the two matrix networks (i.e., the optical signals of the second array and the optical signals of the fourth array) are coherently detected to obtain the dot product operation results of the two calculation results, which are output through a coherent detector. As a result, after the signal to be processed enters the photon neural network until it is output through the light receiving unit, the matrix multiplication and dot product operations can be completed without optical-electrical-optical conversion (i.e., without electrical domain conversion) during the calculation process, which effectively overcomes the problems of high power consumption and rate limitation of the existing neural network technology solutions.

[0024] Furthermore, the calculation results of the photonic neural network described in this embodiment are output using a coherent detector array (i.e., a plurality of coherent detection units arranged in an array), which facilitates subsequent advanced packaging with an integrated circuit.

[0025] Furthermore, the first photon matrix calculation unit and the second photon matrix calculation unit both adopt a "well"-shaped matrix calculation architecture to achieve phase-independent amplitude weight matrix control. The optical amplitude modulation unit can adopt a Mach-Zehnder interferometer (MZI) amplitude controller, specifically using a push-pull drive method to achieve phase-independent amplitude modulation. Furthermore, the phase difference of different paths is compensated by a phase shifter to ensure that the optical signals output by the two photon matrix calculation units are correctly coherently detected at the dot product calculation unit.

[0026] Furthermore, the input of the initial optical signal (for example, the transmission of the optical signal of the first array and the optical signal of the third array) and the actual operation (for example, matrix calculation, dot product calculation) are implemented on the silicon waveguide, and the transmission of the calculation result (for example, the optical signal of the second array and the optical signal of the fourth array) is implemented on the silicon nitride waveguide. The silicon waveguide and the silicon nitride waveguide are located in different waveguide layers (for example, the calculation of the optical signal is located in the first waveguide layer, and the transmission of the calculation result of the optical signal is located in the second waveguide layer. The two waveguide layers can be different layers made of the same material, or different layers made of different materials, such as a silicon waveguide layer and a silicon nitride waveguide layer prepared by a silicon nitride on silicon process) to avoid signal crosstalk affecting the calculation results. Therefore, the photonic neural network described in this embodiment adopts the silicon nitride on silicon process for layout, with high degree of freedom and convenient expansion.

[0027] Furthermore, this embodiment provides a universal optical computing network, in which the calculation results of a certain area (for example, the first photon matrix computing unit or the second photon matrix computing unit) can be output separately to meet the computing requirements of the convolutional neural network. Therefore, the photon neural network described in this embodiment can support multiple types of neural network models, for example, both convolutional neural network models and Transformer neural network models. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic diagram of a photonic neural network according to an embodiment of the present invention;

[0029] Figure 2 yes Figure 1 A schematic diagram of a specific implementation of the illustrated embodiment;

[0030] Figure 3 yes Figure 2A schematic diagram of a specific implementation of a coherent detection unit;

[0031] Figure 4 Is adopted Figure 2 The inference accuracy test results of the Transformer neural network obtained by photonic neural network training are shown. DETAILED DESCRIPTION

[0032] As mentioned in the background technology, the current large models for implementing artificial intelligence algorithms are generally implemented using Transformer neural networks. Transformer networks capture the global dependencies between features based on the self-attention mechanism. Specifically, the self-attention mechanism can be roughly described as:

[0033] 1. Project (i.e., encode) the input sequence into three different vectors (based on matrix representation): query vector Q = XW Q , key vector K = XW K , value vector V = XW V , where X is the feature in the input sequence, W Q , W K , W V are weights respectively; 2. Calculate the similarity between the query vector and the key vector Attention(Q,K,V) based on the following formula:

[0034]

[0035] Among them, the softmax function is used to normalize the numerical vector into a probability distribution vector; d k is the number of columns of the Q, K matrices, that is, the vector dimension; T is the transpose of the matrix.

[0036] Generally speaking, matrix multiplication and addition operations in convolutional neural networks account for 90% of the total amount of operations, while data matrix multiplication and addition operations in the self-attention mechanism account for 30% of the total amount of operations, and data dot product operations account for 60% of the total amount of operations. It can be seen that the traditional optical neural network architecture used for convolution calculations can only undertake a small part of the operations of the Transformer neural network model. As for the data dot product operation, it is either implemented in the electrical domain through electrical ICs, or the outer product calculation of matrices Q and K is completed at a very low calculation frequency in the convolutional optical neural network architecture, and it takes a long time to complete all operations.

[0037] On the other hand, in order to simulate the memory capacity of neural networks in the optical domain and thus realize the execution of sequential tasks, people have developed photonic reservoir networks, which are considered to be a variant of recurrent neural networks. There are two ideas when using optoelectronic devices to build recurrent neural networks. One is a parallel structure, that is, each node of the recurrent neural network is built with an optoelectronic device. The advantage of this idea is that it is more intuitive and has a faster calculation speed due to parallel computing. The other is a serial structure, that is, the original many nodes of the reservoir in this structure are realized by delaying a single nonlinear node. However, the reservoir network only trains the output layer, and does not adjust the connection weights between the nodes in the network, so it performs poorly in processing high-dimensional data and complex information.

[0038] In addition to feedforward neural networks and recurrent neural networks, which are spatially coded artificial neural networks based on computational science, in recent years, spiking neural networks (Spiking Neuron Networks, referred to as SNN) are a network model that is closest to biological neural networks. They use spiking neurons as basic units and can imitate the information encoding and processing process of the human brain. Spiking neurons can be described by a set of differential equations. Compared with other neuron models, they are more complex and more suitable for implementation in hardware rather than software. SNN has the ability to process spatiotemporal data and has the advantage of low energy consumption, which is a major development trend in the future. The application scenarios of SNN include target detection, speech processing, and autonomous driving. The main obstacle to the practical application of photon spiking neural networks is the lack of reliable spiking neural network training algorithms. Therefore, the development of photon spiking neural networks is highly dependent on the development of algorithms. With the development of SNN training algorithms, SNN will usher in its rapid development period.

[0039] In summary, the introduction of photon computing power and the acceleration of optoelectronic fusion artificial intelligence computing have received a lot of attention and importance. However, at present, there is no universal optoelectronic fusion artificial intelligence chip that can support multiple types of neural network models at the same time. At the same time, the existing optoelectronic fusion artificial intelligence chips have problems such as poor reconfigurability, insufficient scalability, and computing energy efficiency far below the theoretical value, making it difficult to apply them on a large scale. Therefore, the development of universal optoelectronic fusion chips is highly anticipated.

[0040] In order to solve at least one of the above technical problems, this embodiment provides a photonic neural network, which is driven by the Transformer neural network algorithm architecture. By constructing two matrix networks (i.e., the first photon matrix calculation unit and the second photon matrix calculation unit), the input signal (i.e., the optical signal of the first array and the optical signal of the third array) is respectively encoded to obtain the corresponding query (query, q), key (key, k) and information extraction value (value, v). Afterwards, the output results of the two matrix networks (i.e., the optical signal of the second array and the optical signal of the fourth array) are coherently detected to obtain the dot product operation result of the two calculation results, and output through a coherent detector. Therefore, after the signal to be processed enters the photonic neural network until it is output through the light receiving unit, the matrix multiplication and dot product operations can be completed without optical-electrical-optical conversion (i.e., without electrical domain conversion) in the intermediate calculation process, which effectively overcomes the problems of high power consumption and rate limitation of the existing neural network technical solutions.

[0041] Furthermore, this embodiment provides a universal optical computing network, in which the calculation results of a certain area or unit (for example, the first photon matrix calculation unit or the second photon matrix calculation unit) can be output separately to meet the calculation requirements of the convolutional neural network. Therefore, the photon neural network described in this embodiment can support multiple types of neural network models, for example, both convolutional neural network models and Transformer neural network models.

[0042] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0043] Figure 1 It is a schematic diagram of a photonic neural network 1 according to an embodiment of the present invention.

[0044] This embodiment can be applied to application scenarios such as image processing, image recognition, speech recognition, gene sequencing, quantum communication or quantum computing, and the photonic neural network 1 described in this embodiment can realize the recognition of images, videos, speech, etc. with better computing power. For example, based on the photonic neural network 1, the Transformer neural network model can be used to recognize images, videos, speech, etc.

[0045] Specifically, refer to Figure 1 In this embodiment, the photonic neural network 1 may include a first photonic matrix calculation unit 11, which is used to perform matrix calculation on the optical signal of the first array to obtain the optical signal of the second array. The matrix calculation at least includes a matrix multiplication operation, and the optical signal of the first array is modulated according to the signal to be processed.

[0046] More specifically, the optical signal of the first array may include at least one optical signal, wherein each optical signal carries information of the signal to be processed. Further, the signal to be processed may include a digital signal or an analog signal to be processed, for example, an image signal for image recognition, a voice signal for voice recognition, etc.

[0047] In a specific implementation, the first photon matrix calculation unit 11 may specifically include an optical modulation unit (referred to as a first optical modulation unit 111 to be distinguished from the second optical modulation unit 121 in the second photon matrix calculation unit 12 mentioned below) for modulating the signal to be processed into light. The signal to be processed may be characterized as an electrical signal carrying corresponding information, and by transmitting the electrical signal to the first optical modulation unit 111, the first optical modulation unit 111 can load the information in the electrical signal onto the optical signal to form an optical signal of the first array.

[0048] Furthermore, the electrical signal transmitted to the first optical modulation unit 111 may be an analog signal.

[0049] In some embodiments, the analog signal may be data to be processed directly obtained from outside the photonic neural network 1. For example, the data to be processed may be an image signal for image recognition represented by an analog signal, or a speech signal for speech recognition. The first optical modulation unit 111 encodes the analog signal into an optical signal for subsequent operation.

[0050] For example, the photonic neural network 1 can receive a voice signal from a microphone, and process the received voice signal to recognize the content of the voice signal. Thus, voice recognition can be implemented based on the photonic neural network 1.

[0051] For another example, the photonic neural network 1 can receive, for example, an image signal from an image acquisition device, and process the received image signal to identify the content of the image signal. Thus, image recognition can be implemented based on the photonic neural network 1. Accordingly, the signal to be processed can be the grayscale of multiple pixels of the image.

[0052] In some embodiments, the analog signal can be converted from a digital signal, where the digital signal is data to be processed input from outside the photonic neural network 1.

[0053] For example, the photonic neural network 1 may include a first electrical processing unit (not shown) for performing digital-to-analog conversion on the data to be processed to generate a corresponding analog signal. Further, the generated analog signal is transmitted to the first optical modulation unit 111 as a signal to be processed.

[0054] Furthermore, the first electrical processing unit may be integrated into the photonic neural network 1. Alternatively, the first electrical processing unit and the photonic neural network 1 may be separately arranged, and the two may be coupled to each other for data transmission.

[0055] In one specific implementation, reference Figure 1 and 2 , the photonic neural network 1 may also include a light source 15 and a beam splitter a1, which are used to generate an optical signal array, each optical signal in the optical signal array includes continuous light of the same or different wavelengths. For example, the beam splitter a1 may be a 1×m beam splitter, which is used to split a single optical signal output by the light source 15 into m optical signals, where m is a positive integer. The result of the beam splitting is an optical signal array. The beam splitter a1 may be a single beam splitting device, or it may be composed of a plurality of beam splitters cascaded. The beam splitter a1 may use a directional coupler (DC for short), a multi-mode interferometer (MMI for short), and the like.

[0056] Further, the first optical modulation unit 111 may include an optical modulator array, coupled to the output end of the beam splitter a1 to receive the optical signal array. For example, the optical modulator array includes a plurality of optical modulators 114, and the plurality of optical modulators 114 correspond one-to-one to the optical signals in the optical signal array to modulate the corresponding optical signals. The optical modulator 114 may be an electro-optical, acousto-optical, thermo-optical, or other type of modulator, such as a differential drive modulator.

[0057] The first optical modulation unit 111 applies a modulation signal to the optical signal array to obtain an optical signal of the first array, wherein the modulation signal is associated with the signal to be processed. The modulation signal is associated with the signal to be processed, which may mean that the specific value of the modulation signal is determined according to the specific information of the signal to be processed.

[0058] For example, after the optical signal array passes through the first optical modulation unit 111, the optical modulator array can be used to adjust the intensity. Depending on the applied voltage, the intensity adjustment amount of each optical signal is different. The voltage applied by the optical modulator 114 and the intensity of the output light are related, and the information of the signal to be processed can be loaded on the light through this relationship.

[0059] In some embodiments, the optical signals of the first array may be optical signals with the same modulation signal, that is, the first optical modulation unit 111 applies the same modulation signal to each beam of optical signal in the optical signal array.

[0060] In a variation, the modulator array may apply different modulation signals to different optical signals in the optical signal array. For example, the modulator array may include m modulators, each of which is used to modulate one optical signal. Specifically, the optical signal loaded by each modulator is the same, but each modulator applies a different voltage to the optical signal, thereby obtaining modulation results of different intensities. As a result, in the optical signals of the first array output by the modulator array, the information to be processed carried by the same optical signal is the same, and the information carried by different optical signals may be different.

[0061] Of course, the modulator array may also apply the same modulation signal to some of the optical signal beams, and the information carried by these different optical signal beams may also be the same.

[0062] In a specific implementation, continue to refer to Figure 1 and Figure 2 , the first photon matrix calculation unit 11 can use a "well"-shaped matrix calculation architecture to perform matrix calculation on the optical signals of the first array. Specifically, the first photon matrix calculation unit 11 may include: m rows of input waveguides and n columns of output waveguides, where n is a positive integer; a first optical amplitude modulation unit array 112, including a plurality of optical amplitude modulation units 113 arranged in an array, for receiving the optical signals of the first array and generating the optical signals of the second array.

[0063] Furthermore, the m rows of input waveguides and the n columns of output waveguides have a plurality of (i.e., m×n) intersections 115, and the plurality of optical amplitude modulation units 113 correspond one-to-one to the plurality of intersections 115. For each of the m×n optical amplitude modulation units 113, an input end of the optical amplitude modulation unit 113 is coupled to an input waveguide of the corresponding intersection 115, and an output end of the optical amplitude modulation unit 113 is coupled to an output waveguide of the corresponding intersection 115.

[0064] Figure 2 In the figure, m=n=3 is used as an example for exemplary display. Nine optical amplitude modulation units 113 are arranged in an array of three rows and three columns. The input waveguide of each row is coupled to the optical modulator 114 of the corresponding row, and the output waveguide of each column is coupled to the dot product calculation unit 13. The input end of the optical amplitude modulation unit 113 located in the same row is coupled to the input waveguide located between the corresponding intersection 115 and the optical modulator 114 of the row. The output end of the optical amplitude modulation unit 113 located in the same column is coupled to the output waveguide located between the corresponding intersection 115 and the dot product calculation unit 13.

[0065] Furthermore, the input waveguide and the output waveguide are not optically coupled at the intersection 115. That is, before the optical signal reaches the intersection 115, it is coupled from the input waveguide to the output waveguide via the corresponding optical amplitude modulation unit 113. The flow path of the optical signal in the first optical amplitude modulation unit array 112 is shown by the dot-dash line with an arrow. Figure 2 In the figure, the intersection 115 marked by the hollow circle represents that there is no optical coupling at this location, that is, there is no optical crosstalk between the waveguides at this location; the intersection of multiple waveguides without hollow circles (such as Figure 2 There is optical coupling at the position b1 in the middle, that is, light can be transferred from one waveguide (e.g., input waveguide) to another waveguide (e.g., output waveguide) at this location.

[0066] In a specific implementation, the transmission of optical signals between the units in the photonic neural network 1 described in this embodiment can be based on waveguide carrying. Specifically, waveguide refers to any structure that can guide optical signals in any way, such as optical fiber, semiconductor waveguide manufactured in a substrate, photonic crystal structure configured to guide optical signals, or any other suitable structure.

[0067] Furthermore, the input waveguide and the output waveguide may be made of different materials. For example, the input waveguide may be a silicon waveguide, and the output waveguide may be a silicon nitride waveguide. Figure 2 In the figure, the red line represents the silicon waveguide and the black line represents the silicon nitride waveguide.

[0068] Furthermore, the input waveguide and the output waveguide can be located in different waveguide layers. For example, the photonic neural network 1 can be manufactured using a semiconductor process. During the preparation process, a silicon waveguide can be prepared on a silicon substrate as an input waveguide, and then a silicon nitride layer is deposited on the silicon substrate, and a silicon nitride waveguide is prepared on the silicon nitride layer as an output waveguide. Furthermore, the silicon waveguide and the silicon nitride waveguide can be located in the upper and lower layers of the semiconductor structure, respectively, and most of the areas between the two layers are isolated from each other (corresponding to the area where the intersection 115 is located), and through holes can also be opened in local areas between the two layers to achieve optical coupling (corresponding to the location of the intersection b1).

[0069] In a specific implementation, the optical amplitude modulation unit 113 may be, for example, a 1-input 1-output (ie, 1×1) MZI controller, configured to perform a multiplication operation on the input signal (ie, the optical signal of the first array) and the weight. The corresponding calculation formula may be, for example: q u =Wq u ·X1,q v =Wq v ·X2,q z =Wq z ·X3, “·” indicates multiplication, u is from 1, 4 and 7, v is from 2, 5 and 8, z is from 3, 6 and 9, q is from u is the calculation result of the optical amplitude modulation unit 113 of the first row, q v is the calculation result of the optical amplitude modulation unit 113 of the second row, q zis the calculation result of the optical amplitude modulation unit 113 in the third row, X1 is the input signal of the first row (i.e., the optical signal of the first array of the optical amplitude modulation unit 113 in the first row), X2 is the input signal of the second row (i.e., the optical signal of the first array of the optical amplitude modulation unit 113 in the second row), X3 is the input signal of the third row (i.e., the optical signal of the first array of the optical amplitude modulation unit 113 in the third row), Wq u is the weight of the optical amplitude modulation unit 113 of the first row, Wq v is the weight of the optical amplitude modulation unit 113 in the second row, Wq z is the weight of the optical amplitude modulation unit 113 in the third row.

[0070] Furthermore, the weights configured for each of the plurality of optical amplitude modulation units 113 may be the same. For example, the weights for each column of optical amplitude modulation units 113 may be the same, such as Wq1=Wq2=Wq3.

[0071] Alternatively, the weight of at least one optical amplitude modulation unit 113 may be different from the weights of other optical amplitude modulation units 113. For example, the weights of the optical amplitude modulation units 113 in each row may be different, such as Wq1≠Wq4≠Wq7.

[0072] In some embodiments, the calculation results of the multiple optical amplitude modulation units 113 corresponding to the same column of output waveguides can be output from the output waveguide in a time-division manner. For example, q1, q2, and q3 can be time-division multiplexed on the same output waveguide. In other words, q1, q2, and q3 can be transmitted from the output waveguide to the dot product calculation unit 13 in sequence.

[0073] Thus, the optical amplitude modulation unit 113 adopts a push-pull driving mode to achieve phase-independent amplitude modulation. Further, multiple optical amplitude modulation units 113, m input waveguides and n output waveguides together construct a "well" matrix computing architecture to achieve phase-independent amplitude weight matrix control.

[0074] In a variation, the optical amplitude modulation unit 113 may also be a microring resonator.

[0075] In a variation, the calculation results of the multiple optical amplitude modulation units 113 corresponding to the same column of output waveguides can be accumulated and output from the output waveguide. For example, q1, q2 and q3 are added to obtain a total calculation result, which is then output to the dot product calculation unit 13 via the output waveguide. Thus, the first photon matrix calculation unit 11 can implement matrix multiplication and addition operations (i.e., multiplication and addition operations).

[0076] In a specific implementation, continue to refer to Figure 2The first photon matrix calculation unit 11 may further include a plurality of phase shifters 116 arranged in an array, corresponding one to one with the plurality of optical amplitude modulation units 113 , and for each phase shifter 116 , the phase shifter 116 is coupled to an output end of the corresponding optical amplitude modulation unit 113 .

[0077] Specifically, the phase shifter 116 is used to correct the calculation result of the optical amplitude modulation unit 113 to compensate for the phase difference of different paths, for example, to ensure that the optical signal transmitted to the second array of the dot product calculation unit 13 via the output waveguide does not generate a phase difference due to the difference in the length of the transmission path.

[0078] Furthermore, the phase shifter 116 and the optical amplitude modulation unit 113 are both connected to the silicon waveguide to perform matrix calculation and phase correction on the silicon waveguide. Furthermore, the output end of the phase shifter 116 is coupled to the silicon nitride waveguide to transmit the calculation result (i.e., the optical signal of the second array) to the outside through the silicon nitride waveguide.

[0079] In a specific implementation, continue to refer to Figure 1 and Figure 2 The photonic neural network 1 may further include: a second photonic matrix calculation unit 12, which is used to perform matrix calculation on the optical signal of the third array to obtain the optical signal of the fourth array. The matrix calculation at least includes a matrix multiplication operation, and the optical signal of the fourth array is modulated according to the signal to be processed.

[0080] Specifically, the signal to be processed modulated to the first photon matrix calculation unit 11 may be the same as the signal to be processed modulated to the second photon matrix calculation unit 12. Therefore, when the Transformer neural network model is implemented based on the photon neural network 1, the query vector and the key vector of the signal to be processed can be encoded in parallel.

[0081] Furthermore, the structure of the second photon matrix calculation unit 12 is similar to that of the first photon matrix calculation unit 11. For example, the second photon matrix calculation unit 12 may also include an optical modulation unit (referred to as a second optical modulation unit 121) and a second optical amplitude modulation unit array 122. The specific structure of the second optical modulation unit 121 may refer to the relevant description of the first optical modulation unit 111 in the above embodiment, and the specific structure of the second optical amplitude modulation unit array 122 may refer to the relevant description of the first optical amplitude modulation unit array 112 in the above embodiment. Figure 2 As shown in the drawing method, the first photon matrix calculation unit 11 has rows in the horizontal direction and columns in the vertical direction; while the second photon matrix calculation unit 12 has columns in the horizontal direction and rows in the vertical direction.

[0082] That is, the optical signal array is modulated by the modulator array to obtain the optical signal of the third array, and the optical amplitude modulation unit 113 multiplies the input signal (i.e., the optical signal of the third array) by the weight, and the phase is corrected by the phase shifter 116, and the calculation result is coupled to the silicon nitride waveguide. The calculation structures in the same column can be output sequentially or added to complete the multiplication and addition operation and then output to the dot product calculation unit 13. The corresponding calculation formula can be, for example: k u =Wk u ·X1,k v =Wk v ·X2,k z =Wk z X3, u is from 1, 4 and 7, v is from 2, 5 and 8, z is from 3, 6 and 9, k u is the calculation result of the optical amplitude modulation unit 113 of the first row, k v is the calculation result of the optical amplitude modulation unit 113 of the second row, k z is the calculation result of the optical amplitude modulation unit 113 in the third row, X1 is the input signal of the first row (i.e., the optical signal of the first array of the optical amplitude modulation unit 113 in the first row), X2 is the input signal of the second row (i.e., the optical signal of the first array of the optical amplitude modulation unit 113 in the second row), X3 is the input signal of the third row (i.e., the optical signal of the first array of the optical amplitude modulation unit 113 in the third row), Wk u is the weight of the optical amplitude modulation unit 113 of the first row, Wk v is the weight of the optical amplitude modulation unit 113 in the second row, Wk z is the weight of the optical amplitude modulation unit 113 in the third row.

[0083] Furthermore, the weights configured for each of the plurality of optical amplitude modulation units 113 may be the same. For example, the weights for each column of optical amplitude modulation units 113 may be the same, such as Wk4=Wk5=Wk6.

[0084] Alternatively, the weight of at least one optical amplitude modulation unit 113 may be different from the weights of other optical amplitude modulation units 113. For example, the weights of the optical amplitude modulation units 113 in each row may be different, such as Wk1≠Wk4≠Wk7.

[0085] Furthermore, the weights of the optical amplitude modulation units 113 in the first photon matrix calculation unit 11 may be independent of the weights of the optical amplitude modulation units 113 in the second photon matrix calculation unit 12. That is, the weights of the optical amplitude modulation units 113 in the two photon matrix calculation units may be set separately.

[0086] In a typical application scenario, refer to Figure 1 and Figure 2The light generated by the light source 15 enters the first photon matrix calculation unit 11 and the second photon matrix calculation unit 12 respectively through the beam splitter a2.

[0087] In the first photon matrix calculation unit 11, the light beam input by the beam splitter a2 is further divided into m beams of light via the beam splitter a1, wherein each beam of light is loaded with a signal to be processed by the corresponding optical modulator 114 in the modulator array, thereby obtaining an optical signal of the first array. Further, the optical signal of the first array is input into each optical amplitude modulation unit 113 via a silicon waveguide, and the optical amplitude modulation unit 113 performs matrix calculation on the optical signal and outputs the calculation result, and the calculation result is corrected via a phase shifter 116 to obtain an optical signal of the second array. Further, the optical signal of the second array is coupled to a silicon nitride waveguide, and output to the dot product calculation unit 13 via the silicon nitride waveguide.

[0088] Similarly, in the second photon matrix calculation unit 12, the light beam input by the beam splitter a2 is further divided into m beams of light via the beam splitter a1, wherein each beam of light is loaded with a signal to be processed by the corresponding optical modulator 114 in the modulator array, thereby obtaining an optical signal of the third array. Further, the optical signal of the third array is input into each optical amplitude modulation unit 113 via a silicon waveguide, and the optical amplitude modulation unit 113 performs matrix calculation on the optical signal and outputs the calculation result, and the calculation result is corrected via a phase shifter 116 to obtain an optical signal of the fourth array. Further, the optical signal of the fourth array is coupled to a silicon nitride waveguide, and output to the dot product calculation unit 13 via the silicon nitride waveguide.

[0089] In a specific implementation, continue to refer to Figure 1 The photonic neural network 1 may further include a dot product calculation unit 13, which is respectively coupled to the first photon matrix calculation unit 11 and the second photon matrix calculation unit 12, and the dot product calculation unit 13 is used to perform dot product calculation on the optical signal of the second array and the optical signal of the fourth array to obtain the optical signal of the fifth array.

[0090] Specifically, the first optical amplitude modulation unit array 112 and the second optical amplitude modulation unit array 122 can both be arranged in an m×n array. Accordingly, the dot product calculation unit 13 performs a dot product operation on the n q values ​​output by the first optical amplitude modulation unit array 112 and the n k values ​​output by the second optical amplitude modulation unit array 122. Figure 2 The example of m=n is used for exemplary display. In practical applications, the values ​​of m and n may not be equal. The first optical amplitude modulation unit array 112 and the second optical amplitude modulation unit array 122 may also independently set their respective m and n, as long as the number of q values ​​output by the first optical amplitude modulation unit array 112 is equal to the number of k values ​​output by the second optical amplitude modulation unit array 122. For example, the first optical amplitude modulation unit array 112 may have 3 rows and 2 columns, and the second optical amplitude modulation unit array 122 may have 2 rows and 2 columns.

[0091] Further reference Figure 2 The dot product calculation unit 13 may include: n columns of first waveguides 131, for receiving the optical signal of the second array; n rows of second waveguides 132, for receiving the optical signal of the fourth array; a plurality of coherent detection units 133 arranged in an array, respectively coupled to the n columns of first waveguides 131 and the n rows of second waveguides 132, to perform dot product calculation on the optical signal of the second array and the optical signal of the fourth array to obtain the optical signal of the fifth array.

[0092] Further, the n columns of first waveguides 131 can be coupled to the n columns of output waveguides of the first optical amplitude modulation unit array 112 in a one-to-one correspondence. Similarly, the n rows of second waveguides 132 can be coupled to the n columns of output waveguides of the second optical amplitude modulation unit array 122 in a one-to-one correspondence. In some embodiments, the first waveguide 131 can reuse the output waveguide of the corresponding first optical amplitude modulation unit array 112, and similarly, the second waveguide 132 can reuse the output waveguide of the corresponding second optical amplitude modulation unit array 122. For example, a silicon nitride waveguide can be prepared in a silicon nitride waveguide layer, a portion of the silicon nitride waveguide is optically coupled to each optical amplitude modulation unit 113, and the remaining portion of the silicon nitride waveguide is optically coupled to the coherent detection unit 133.

[0093] Furthermore, the material of the first waveguide 131 and the second waveguide 132 may be the same as that of the output waveguide, for example, both may be silicon nitride waveguides.

[0094] Furthermore, the n columns of first waveguides 131 and the n rows of second waveguides 132 have a plurality of (e.g., n×n) intersections 115, and the plurality of intersections 115 correspond one-to-one to the plurality of coherent detection units 133. For each coherent detection unit 133, a first input terminal c1 of the coherent detection unit 133 is coupled to the first waveguide 131 of the corresponding intersection 115, and a second input terminal c2 of the coherent detection unit 133 is coupled to the second waveguide 132 of the corresponding intersection 115.

[0095] Still Figure 2As shown, the first photon matrix calculation unit 11 includes 3 output waveguides, and the second photon matrix calculation unit 12 includes 3 output waveguides. At time t1, the three first waveguides 131 respectively receive the calculation results {q1, q4, q7} output by the first photon matrix calculation unit 11, and the three second waveguides 132 respectively receive the calculation results {k1, k4, k7} output by the second photon matrix calculation unit 12. Then, the nine coherent detection units 133 included in the dot product calculation unit 13 can perform matrix dot product operations on {q1, q4, q7} and {k1, k4, k7} to obtain the optical signal of the fifth array. At time t2, the three first waveguides 131 respectively receive the calculation results {q2, q6, q8} output by the first photon matrix calculation unit 11, and the three second waveguides 132 respectively receive the calculation results {k3, k5, k7} output by the second photon matrix calculation unit 12. Then, the nine coherent detection units 133 included in the dot product calculation unit 13 can perform matrix dot product operations on {q2, q6, q8} and {k3, k5, k7} to obtain the optical signal of the fifth array.

[0096] Further, the first waveguide 131 and the second waveguide 132 are not optically coupled at the intersection 115. That is, the optical signal of the second array transmitted via the first waveguide 131 and the optical signal of the fourth array transmitted via the second waveguide 132 are both transmitted on their respective waveguides without crosstalk.

[0097] In some embodiments, the first waveguide 131 and the second waveguide 132 may be located in the same waveguide layer. For example, the first waveguide 131 and the second waveguide 132 are both silicon nitride waveguides. Furthermore, the first waveguide 131 and the second waveguide 132 are located at different heights in the same waveguide layer, and there is no crosstalk of optical signals between the two waveguides.

[0098] In some embodiments, for each coherent detection unit 133, reference Figure 3 The coherent detection unit 133 may include: a 2×2 optical amplitude modulation unit 134 and a balanced detector 135. The 2×2 optical amplitude modulation unit 134 may be a 2×2 MMI.

[0099] Specifically, the two input ends of the 2×2 optical amplitude modulation unit 134 are respectively coupled to the first waveguide 131 and the second waveguide 132. For example, the first input end c1 is coupled to the first waveguide 131, and the second input end 132 is coupled to the second waveguide 132. Further, the two output ends (denoted as c3 and c4) of the 2×2 optical amplitude modulation unit 134 are respectively coupled to the balanced detector 136.

[0100] Furthermore, the output end of the balanced detector 136 is coupled to the light receiving unit 14 .

[0101] In a specific implementation, the first waveguide 131 and the coherent detection unit 133 may be located in different waveguide layers. Similarly, the second waveguide 132 and the coherent detection unit 133 may also be located in different waveguide layers. For example, the first waveguide 131 and the second waveguide 132 are located in a silicon nitride waveguide layer, and the coherent detection unit 133 is located in a silicon waveguide layer.

[0102] That is, in this embodiment, for at least one of the matrix calculation and the dot product calculation, the calculation of the optical signal and the transmission of the calculation result of the optical signal are performed in different waveguide layers. For example, the optical signal is calculated in the silicon waveguide layer, and the calculation result obtained in the silicon waveguide layer is coupled to the silicon nitride waveguide layer and transmitted in the silicon nitride waveguide layer. Figure 2 and Figure 3 In the figure, different colors are used to identify different components / functional modules, where red identifies devices on the silicon waveguide layer (such as silicon waveguide), and black identifies devices on the silicon nitride waveguide layer (such as silicon nitride waveguide).

[0103] Furthermore, the optical interference unit 113, the 2×2 optical amplitude modulation unit 134 and the phase shifter 116 are all located in the silicon waveguide layer, so that the optical signal can be calculated in the silicon waveguide layer with higher calculation efficiency and lower loss. The silicon nitride waveguide layer preferably forms an output waveguide, a first waveguide 131 and a second waveguide 132 to transmit the calculation results of each unit.

[0104] Furthermore, the dot product calculation unit 13 may also include a plurality of vertical coupling structures (not shown), corresponding one to one with the plurality of coherent detection units 133. For each vertical coupling structure, the vertical coupling structure is used to couple the optical signal transmitted by the first waveguide 131 and / or the second waveguide 132 to the waveguide layer where the corresponding coherent detection unit 133 is located.

[0105] For example, a vertical coupling structure is provided between the first input terminal c1 and the first waveguide 131, and the optical signal of the second array transmitted by the first waveguide 131 (silicon nitride waveguide) is transmitted to the first input terminal c1 via the vertical coupling structure and then enters the silicon waveguide layer. For another example, a vertical coupling structure is provided between the second input terminal c2 and the second waveguide 132, and the optical signal of the fourth array transmitted by the second waveguide 132 (silicon nitride waveguide) is transmitted to the second input terminal c2 via the vertical coupling structure and then enters the silicon waveguide layer.

[0106] In a typical application scenario, the calculation results of the first optical amplitude modulation unit array 112 (for example, the query vector q) and the calculation results of the second optical amplitude modulation unit array 122 (for example, the key vector k) are output to the dot product calculation unit 13, and the 1 / n light is coupled to the silicon waveguide layer through the vertical coupling structure, and respectively input into the two input ports of the corresponding 2×2MMI (i.e., the first input port c1 and the second input port c2), and the interference result is input into the two input ports of the balanced detector 135 through the two output ports of the 2×2MMI (i.e., the first output port c3 and the second output port c4), completing the dot product operation of q and k. The corresponding calculation formula is: Among them, i is the number of columns, j is the number of rows, and q i is the calculated result of the transmission of the first waveguide 131 in the i-th column (i.e., the optical signal of the second array), k j is the calculated result of the transmission of the second waveguide 132 in the jth row (i.e., the optical signal of the fourth array), is the phase difference between the optical signal of the second array and the optical signal of the fourth array.

[0107] Taking the dot product calculation of {q1, q4, q7} and {k1, k4, k7} as an example, when the optical signal carrying the calculation result q1 transmitted by the first waveguide 131 in the first column passes through the three coherent detection units 133 in the first column, 1 / 3 of the optical signal is coupled to the silicon waveguide layer where the corresponding coherent detection unit 133 is located through three vertical coupling structures. Similarly, when the optical signal carrying the calculation result k1 transmitted by the second waveguide 132 in the first row passes through the three coherent detection units 133 in the first row, 1 / 3 of the optical signal is coupled to the silicon waveguide layer where the corresponding coherent detection unit 133 is located through three vertical coupling structures. By analogy, each coherent detection unit 133 receives the optical signal from the first photon matrix calculation unit 11 and the second photon matrix calculation unit 12, respectively.

[0108] Further, the first row and first column coherent detection unit 133 performs a dot product operation on q1 and k1, the first row and second column coherent detection unit 133 performs a dot product operation on q4 and k1, the first row and third column coherent detection unit 133 performs a dot product operation on q7 and k1, the second row and first column coherent detection unit 133 performs a dot product operation on q1 and k4, and the third row and first column coherent detection unit 133 performs a dot product operation on q1 and k7. By analogy, the dot product calculation of the two matrices {q1, q4, q7} and {k1, k4, k7} is implemented based on 9 coherent detection units 133.

[0109] In some embodiments, Δφ can be adjusted to an integer multiple of 2π by the phase shifter 136, so as to ensure that the output result is: m ij =2q i k jFor example, a phase shifter 136 may be coupled between the first input terminal c1 and the 2×2 optical amplitude modulation unit 134 to adjust the phase of the optical signal of the second array transmitted by the first waveguide 131 .

[0110] In a specific implementation, continue to refer to Figure 1 The photonic neural network 1 may further include: a light receiving unit 14 coupled to the dot product calculation unit 13 to receive the optical signal of the fifth array, and the light receiving unit 14 obtains a processed signal based on the optical signal of the fifth array.

[0111] Specifically, the light receiving unit 14 may include: a light detector for performing photoelectric conversion on the optical signal of the fifth array; a high-pass filtering unit coupled to the light detector, and the high-pass filtering unit performs high-pass filtering on the output of the light detector to obtain the processed signal.

[0112] Specifically, the number of the light detectors can be multiple, and the light detector array composed of multiple light detectors can be coupled to multiple coherent detection units 133 of the dot product calculation unit 13, wherein each light detector is used to process a beam of optical signals in the fifth array of optical signals output by the balanced detector 135 in the corresponding coherent detection unit 133.

[0113] Furthermore, the photonic neural network 1 may also include a second electrical processing unit (not shown), coupled to the light receiving unit 14. The second electrical processing unit may be an analog-to-digital converter, which is used to convert the processed signal represented by the analog signal into a digital signal, thereby obtaining processed data that can be directly read and processed by a computer.

[0114] Furthermore, the processed signal may be amplified and other operations.

[0115] From the above, by adopting this implementation, vector multiplication and dot product operations can be realized in the entire optical domain, so that the computational efficiency and power consumption are higher when the photonic neural network 1 using this implementation is used for Transformer calculation. Specifically, the first photon matrix calculation unit 11 performs a convolution operation on the processed signal to obtain a first convolution operation result (based on the optical signal carried by the second array), and the optical signal of the second array performs a convolution operation on the processed signal to obtain a second convolution operation result (based on the optical signal carried by the fourth array). The first convolution operation result and the second convolution operation result are respectively input as the rows and columns of the matrix into the dot product calculation unit 13 for dot product calculation, and the operation result that includes both feature extraction and dependency between features is obtained (based on the optical signal carried by the fifth array). In this way, an optical computing network architecture that simultaneously accelerates matrix multiplication calculations (for encoding) and dot product calculations (for obtaining dependencies) in the Transformer neural network is realized.

[0116] Furthermore, the calculation results of the photonic neural network 1 described in this embodiment are output using a coherent detector array (i.e., a plurality of coherent detection units 133 arranged in an array), which facilitates subsequent advanced packaging with an integrated circuit.

[0117] Furthermore, the first photon matrix calculation unit 11 and the second photon matrix calculation unit 12 both adopt a "well"-shaped matrix calculation architecture to achieve phase-independent amplitude weight matrix control. The optical amplitude modulation unit 113 can adopt an MZI amplitude controller, specifically using a push-pull drive method to achieve phase-independent amplitude modulation. Furthermore, the phase difference of different paths is supplemented by a phase shifter 116 to ensure that the optical signals output by the two photon matrix calculation units are correctly coherently detected at the dot product calculation unit 13.

[0118] Furthermore, the input of the initial optical signal (for example, the transmission of the optical signal of the first array and the optical signal of the third array) and the actual operation (for example, matrix calculation, dot product calculation) are implemented on the silicon waveguide, and the transmission of the operation result (for example, the optical signal of the second array and the optical signal of the fourth array) is implemented on the silicon nitride waveguide. The silicon waveguide and the silicon nitride waveguide are located in different waveguide layers (for example, different waveguide layers) to avoid signal crosstalk affecting the operation result. Therefore, the photonic neural network 1 described in this embodiment adopts the silicon nitride on silicon process for layout, with high degree of freedom and convenient expansion.

[0119] In a typical application scenario, Figure 2 The photonic neural network 1 shown in the figure is simulated and calculated, and three unit vectors A=|111|, B=|111| are input from the first light modulation unit 111 and the second light modulation unit 121 respectively, and the photonic neural network 1 is constructed into a diagonal matrix, and the weight coefficients are as follows:

[0120] That is, the weight of each optical amplitude modulation unit 113 in the first optical amplitude modulation unit array 112 .

[0121] That is, the weight of each optical amplitude modulation unit 113 in the second optical amplitude modulation unit array 122 .

[0122] The calculation result of the dot product calculation unit 13 is output by the balanced detector function, and the result is as follows:

[0123]

[0124] The output is as expected.

[0125] Furthermore, the accuracy of the photonic neural network 1 when executing the Transformer neural network was trained. The preset data set was processed, 90% of which was extracted for training and the remaining 10% was used for testing. The trained neural network architecture was inferred on the test set, and the inference accuracy was about 97.9%, as shown in Figure 2. Figure 4 shown.

[0126] In a typical application scenario, Figure 2 The photonic neural network 1 and the traditional 3×3 optical neural network architecture shown execute the Transformer algorithm and compare the execution effects. In this application scenario, the input signal of the Transformer algorithm includes 4 features a1, a2, a3, and a4, and the 4 features can be, for example, 4 pixels in an image.

[0127] Using a 3×3 optical neural network, 12 independent calculations are required to obtain the q, k, and v matrices corresponding to a1, a2, a3, and a4, and then the outer product calculation of the matrix q and k is completed at a kilohertz (KHz) calculation frequency. It takes about 10 milliseconds to complete all the operations.

[0128] By using the photonic neural network 1 described in the present embodiment to implement an optical Transformer neural network, all calculations can be completed at 20 GHz at one time, with an estimated time of less than 5ns, and the calculation delay is improved by about three orders of magnitude.

[0129] In a variation of the above embodiment, the light receiving unit 14 may also be coupled to the first photon matrix calculation unit 11 to receive the optical signal of the second array, and the light receiving unit 14 obtains the processed signal based on the optical signal of the second array. Specifically, the processed signal obtained based on the optical signal of the second array is equivalent to performing a convolution operation on the signal to be processed.

[0130] In another variation, the light receiving unit 14 can also be coupled to the second photon matrix calculation unit 12 to receive the optical signal of the fourth array, and the light receiving unit 14 obtains the processed signal based on the optical signal of the fourth array. Specifically, the processed signal obtained based on the optical signal of the fourth array is equivalent to performing a convolution operation on the signal to be processed.

[0131] Therefore, this variation provides a universal optical computing network, in which the calculation results of a certain area (for example, the first photon matrix calculation unit 11 or the second photon matrix calculation unit 12) can be output separately to meet the calculation requirements of the convolutional neural network. Therefore, the photon neural network 1 described in this embodiment can support multiple types of neural network models, for example, both convolutional neural network models and Transformer neural network models.

[0132] From the above, this implementation scheme integrates matrix multiplication and dot product calculation, solving the problems of low computational efficiency, high power consumption, and low integration in the prior art. Furthermore, it is based on the silicon nitride on silicon active process, with high process feasibility; and the unit devices and networks have been verified and are easy to implement; the number of optical-electrical-optical conversions is reduced, fewer electric drive chips are required, the overall network integration is higher, the required chip area is smaller, and the cost is lower.

[0133] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A photonic neural network, characterized in that: include: A first photon matrix calculation unit, configured to perform matrix calculation on the optical signals of the first array to obtain optical signals of the second array, wherein the matrix calculation at least includes a matrix multiplication operation, and the optical signals of the first array are obtained by modulating the signals to be processed; a second photon matrix calculation unit, configured to perform matrix calculation on the optical signal of the third array to obtain the optical signal of the fourth array, wherein the matrix calculation at least includes a matrix multiplication operation, and the optical signal of the fourth array is obtained by modulating the signal to be processed; a dot product calculation unit, coupled to the first photon matrix calculation unit and the second photon matrix calculation unit, respectively, and configured to perform a dot product calculation on the optical signal of the second array and the optical signal of the fourth array to obtain an optical signal of a fifth array; A light receiving unit is coupled to the dot product calculation unit to receive the optical signal of the fifth array, and the light receiving unit obtains a processed signal based on the optical signal of the fifth array.

2. The photonic neural network according to claim 1, characterized in that: The dot product calculation unit comprises: n columns of first waveguides, used for receiving the optical signal of the second array; n rows of second waveguides, used to receive optical signals from the fourth array; A plurality of coherent detection units arranged in an array are respectively coupled to the n columns of first waveguides and the n rows of second waveguides to perform dot product calculation on the optical signal of the second array and the optical signal of the fourth array to obtain the optical signal of the fifth array.

3. The photonic neural network according to claim 2, characterized in that: The n columns of first waveguides and the n rows of second waveguides have multiple intersections, and the multiple intersections correspond one-to-one to the multiple coherent detection units. For each of the coherent detection units, the first input end of the coherent detection unit is coupled to the first waveguide of the corresponding intersection, and the second input end of the coherent detection unit is coupled to the second waveguide of the corresponding intersection.

4. The photonic neural network according to claim 3, characterized in that: The first waveguide and the second waveguide are not optically coupled at the intersection.

5. The photonic neural network according to claim 2, characterized in that: For each of the coherent detection units, the coherent detection unit includes: a 2×2 optical amplitude modulation unit and a balanced detector, the two input ends of the 2×2 optical amplitude modulation unit are respectively coupled to the first waveguide and the second waveguide, the two output ends of the 2×2 optical amplitude modulation unit are respectively coupled to the balanced detector, and the output end of the balanced detector is coupled to the optical receiving unit.

6. The photonic neural network according to claim 2, characterized in that: The first waveguide and the second waveguide are located in the same waveguide layer, and the first waveguide and the coherent detection unit are located in different waveguide layers.

7. The photonic neural network according to claim 6, characterized in that: The dot product calculation unit also includes: A plurality of vertical coupling structures correspond one to one with the plurality of coherent detection units. For each of the vertical coupling structures, the vertical coupling structure is used to couple the optical signal transmitted by the first waveguide and / or the second waveguide to the waveguide layer where the corresponding coherent detection unit is located.

8. The photonic neural network according to claim 1, characterized in that: For any one of the first photon matrix calculation unit and the second photon matrix calculation unit, the photon matrix calculation unit includes: An optical modulation unit, used for modulating a signal to be processed into light; A plurality of optical amplitude modulation units arranged in an array are used to receive an optical signal of an input array and generate an optical signal of an output array, wherein the optical signal of the input array is the optical signal of the first array or the optical signal of the third array, and the optical signal of the output array is the optical signal of the second array or the optical signal of the fourth array.

9. The photonic neural network according to claim 8, characterized in that: The photon matrix calculation unit also includes m rows of input waveguides and n columns of output waveguides, where m and n are both positive integers, and the m rows of input waveguides and the n columns of output waveguides have multiple intersections. The multiple optical amplitude modulation units correspond one-to-one to the multiple intersections. For each of the optical amplitude modulation units, the input end of the optical amplitude modulation unit is coupled to the input waveguide of the corresponding intersection, and the output end of the optical amplitude modulation unit is coupled to the output waveguide of the corresponding intersection.

10. The photonic neural network according to claim 9, characterized in that: The input waveguide and the output waveguide are made of different materials; and / or, the input waveguide and the output waveguide are located in different waveguide layers; and / or, the input waveguide and the output waveguide have no optical coupling at the intersection.

11. The photonic neural network according to claim 9, characterized in that: The calculation results of the multiple optical amplitude modulation units corresponding to the same column of output waveguides are output from the output waveguide in a time-division manner, or the calculation results of the multiple optical amplitude modulation units corresponding to the same column of output waveguides are accumulated and output from the output waveguide.

12. The photonic neural network according to claim 8, characterized in that: The photon matrix calculation unit also includes: The plurality of phase shifters arranged in an array correspond to the plurality of optical amplitude modulation units one by one. For each phase shifter, the phase shifter is coupled to an output end of the corresponding optical amplitude modulation unit.

13. The photonic neural network according to claim 1, characterized in that: For at least one of the matrix calculation and the dot product calculation, the calculation of the optical signal and the transmission of the calculation result of the optical signal are performed in different waveguide layers.

14. The photonic neural network according to claim 13, characterized in that: The optical signal is calculated in the silicon waveguide layer, and the optical signal is transmitted in the silicon nitride waveguide layer.

15. The photonic neural network according to claim 1, characterized in that: The light receiving unit is also coupled to the first photon matrix calculation unit to receive the optical signal of the second array, and the light receiving unit obtains a processed signal based on the optical signal of the second array; And / or, the light receiving unit is further coupled to the second photon matrix calculation unit to receive the optical signal of the fourth array, and the light receiving unit acquires a processed signal based on the optical signal of the fourth array.

16. The photonic neural network according to claim 1, characterized in that: The photonic neural network is used at least for generative artificial intelligence, and the generative artificial intelligence is at least selected from: speech generation, image generation and text generation.

Images