Optoelectronic hybrid computing method and network
By employing a hybrid optoelectronic computing architecture that combines optical and electronic computing, the high computational complexity and high energy consumption of the Transformer model are resolved, enabling real-time target detection with low latency and high energy efficiency, suitable for intelligent driving, industrial inspection, and intelligent security.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-29
- Publication Date
- 2026-04-07
AI Technical Summary
The computational complexity and energy consumption of the Transformer model far exceed the development speed of electronic chips, especially in long sequence tasks, resulting in high latency and high power consumption, making it difficult to meet the low latency and high energy efficiency requirements of real-time target detection.
A hybrid optoelectronic computing architecture is adopted, which uses the Mach-Zehnder interferometer (MZI) optical matrix computing unit for linear transformation calculations and combines it with electronic chips for nonlinear calculations and control logic. The hybrid optoelectronic computing method realizes the calculation of self-attention and feedforward neural networks, reducing power consumption and improving inference efficiency.
While maintaining accuracy comparable to pure electronic computing, it significantly improves inference efficiency and reduces power consumption costs, making it suitable for real-time target detection in fields such as intelligent driving, industrial inspection, and intelligent security.
Smart Images
Figure CN121809567A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of neural networks, and in particular to an optoelectronic hybrid computing method and network. BACKGROUND
[0002] In recent years, the Transformer architecture, a deep learning model based on self-attention mechanism, has become the de facto standard in artificial intelligence, especially in the fields of natural language processing (NLP), computer vision (CV), and speech recognition (ASR). The core self-attention mechanism and feedforward network (FFN) endow the model with excellent representation ability through global interaction and complex transformation, but also introduce unprecedented computational complexity and energy consumption requirements. With the exponential growth of model size (from million parameters to hundred billion parameters), its demand for computing power and energy efficiency has far exceeded the development speed of electronic chips (such as CPU, GPU) based on the Von Neumann architecture following Moore's Law. SUMMARY
[0003] Therefore, the present application aims to provide an optoelectronic hybrid computing method and network to significantly improve inference efficiency and reduce power consumption cost while maintaining comparable accuracy to pure electronic computing in vision Transformer (VIT), which can be widely applied in intelligent driving, industrial detection, and intelligent security fields, and meet the low latency and high energy efficiency requirements of real-time target detection.
[0004] In a first aspect, an embodiment of the present application provides an optoelectronic hybrid computing method, the method comprising: performing electro-optical conversion on an electrical signal of input data to obtain an optical signal, and pre-modulating the optical signal of a fixed weight matrix; performing self-attention mechanism optoelectronic hybrid computing based on the optical signal of the input data and the optical signal of the weight matrix to obtain an optoelectronic hybrid computing result; performing feedforward neural network optoelectronic hybrid computing based on the optoelectronic hybrid computing result and the optical signal of the weight matrix to obtain a feedforward neural network optoelectronic hybrid computing result; and sequentially performing residual connection computing and layer normalization processing on the feedforward neural network optoelectronic hybrid computing result to obtain a final output result.
[0005] In an optional embodiment of this application, the step of performing electro-optic conversion on the electrical signal of the input data to obtain an optical signal includes: loading the input data into a high-bandwidth memory and transmitting the input data to a static random access memory; reading the electrical signal of the input data from the static random access memory; and loading the amplitude or phase information of the electrical signal onto an optical carrier through a digital-to-analog converter and an electro-optic modulator to form an information-carrying light vector as the optical signal of the input data.
[0006] In an optional embodiment of this application, the step of pre-modulating the optical signal of the weight matrix includes: pre-modulating and fixing the electrical signal of the weight matrix in the optical parameters of the Mach-Zehnder interferometer network through singular value decomposition to obtain the optical signal of the weight matrix.
[0007] In an optional embodiment of this application, the aforementioned Mach-Zehnder interferometer is used to construct an optical computing unit, which is used to perform linear transformation calculations in the forward inference of a neural network; wherein, the linear transformation calculations include matrix multiplication and linear transformation.
[0008] In an optional embodiment of this application, the nonlinear calculations and control logic in the neural network are performed via a graphics processing unit platform.
[0009] In an optional embodiment of this application, the step of performing self-attention mechanism optoelectronic hybrid computation based on the optical signal of the input data and the optical signal of the weight matrix to obtain the optoelectronic hybrid computation result includes: generating optical signals of a query vector, a key vector, and a value vector based on the optical signal of the input data and the optical signal of the weight matrix; performing a multiplication operation between the query vector and the transposed key vector through the optical matrix core to obtain the computation result; converting the computation result into an electrical signal through an optoelectronic conversion module and an analog-to-digital converter; and performing processing on the electrical signal of the computation result. The electrical signal of the attention weight is obtained by function operation; the electrical signal of the attention weight is converted into an optical signal by modulator; and the attention weight and value vector are calculated by attention weight multiplication through the core of the optical matrix to obtain the optoelectronic hybrid calculation result.
[0010] In an optional embodiment of this application, the step of generating optical signals of query vector, key vector and value vector based on the optical signal of input data and the optical signal of weight matrix includes: injecting the optical signal of input data into the Mach-Zehnder interferometer grid, performing matrix multiplication through the Mach-Zehnder interferometer grid, and outputting the optical signals of query vector, key vector and value vector.
[0011] In an optional embodiment of the present application, the step of performing the photoelectric hybrid calculation of the light signal based on the photoelectric hybrid calculation result and the light signal of the weight matrix to obtain the photoelectric hybrid calculation result of the feedforward neural network includes: injecting the photoelectric hybrid calculation result into a first optical core configured with the weight matrix to perform optical matrix multiplication calculation to obtain a first linearly transformed light signal; converting the first linearly transformed light signal into an electrical signal through an analog-to-digital converter and a photoelectric detector, performing a GELU function operation on the first linearly transformed electrical signal to obtain a GELU activated electrical signal; converting the GELU activated electrical signal into a light signal through a digital-to-analog converter and an electro-optical detector; and injecting the GELU activated light signal into a second optical core configured with the weight matrix to perform optical matrix multiplication calculation to obtain a second linearly transformed light signal, wherein the second linearly transformed light signal is taken as the photoelectric hybrid calculation result of the feedforward neural network.
[0012] In an optional embodiment of the present application, the step of sequentially performing the residual connection calculation and the layer normalization processing on the photoelectric hybrid calculation result of the feedforward neural network to obtain the final output result includes: converting the photoelectric hybrid calculation result of the feedforward neural network into an electrical signal through a photoelectric detector and an analog-to-digital converter; adding the photoelectric hybrid calculation result and the electrical signal of the photoelectric hybrid calculation result of the feedforward neural network through residual connection to obtain a residual connection calculation result; and performing layer normalization processing on the residual connection calculation result to obtain the final output result.
[0013] In a second aspect, the embodiments of the present application further provide an optical-electric hybrid computing network, which includes: a system initialization module configured to perform electro-optical conversion on an electrical signal of input data to obtain a light signal, and pre-modulate a light signal of a fixed weight matrix; a self-attention mechanism optical-electric hybrid computing module configured to perform self-attention mechanism optical-electric hybrid computation based on the light signal of the input data and the light signal of the weight matrix to obtain an optical-electric hybrid computation result; a feedforward neural network optical-electric hybrid computing module configured to perform feedforward neural network optical-electric hybrid computation based on the optical-electric hybrid computation result and the light signal of the weight matrix to obtain a feedforward neural network optical-electric hybrid computation result; and a final output module configured to sequentially perform residual connection calculation and layer normalization processing on the feedforward neural network optical-electric hybrid computation result to obtain a final output result.
[0014] The embodiments of the present application have the following beneficial effects: The embodiment of the present application provides a kind of optoelectronic hybrid computing method and network, the electrical signal of input data is converted into optical signal by electro-optical conversion, and the optical signal of fixed weight matrix is pre-modulated;Based on the optical signal of input data and the optical signal of weight matrix, self-attention mechanism optoelectronic hybrid computing is carried out, and optoelectronic hybrid computing result is obtained;Based on optoelectronic hybrid computing result and the optical signal of weight matrix, feedforward neural network optoelectronic hybrid computing is carried out, and feedforward neural network optoelectronic hybrid computing result is obtained;Feedforward neural network optoelectronic hybrid computing result is sequentially carried out residual connection calculation and layer normalization processing, and final output result is obtained.In the mode, while maintaining the accuracy equivalent to pure electronic calculation in VIT, the inference efficiency is significantly improved, and the power consumption cost is reduced, which can be widely applied in intelligent driving, industrial detection and intelligent security fields, to meet the low delay and high energy efficiency demand of real-time target detection.
[0015] Other features and advantages of the present disclosure will be set forth in the following description, or can be inferred from the description, or can be determined without doubt, or can be known by implementing the above-mentioned technologies of the present disclosure.
[0016] In order to make the above-mentioned purposes, features and advantages of the present disclosure more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS
[0017] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the description of specific embodiments or prior art. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings without creative labor on the basis of these drawings.
[0018] Figure 1 A flow chart of an optoelectronic hybrid computing method provided by the embodiment of the present application is provided. Figure 2 A structure schematic diagram of an optoelectronic hybrid computing network provided by the embodiment of the present application is provided. Figure 3 A structure schematic diagram of a Mach-Zehnder interferometer provided by the embodiment of the present application is provided. Figure 4 A flow chart of another optoelectronic hybrid computing method provided by the embodiment of the present application is provided. Figure 5 A schematic diagram of self-attention mechanism optoelectronic hybrid computing provided by the embodiment of the present application is provided. Figure 6 A schematic diagram of feedforward neural network optoelectronic hybrid computing provided by the embodiment of the present application is provided. Figure 7A structural schematic diagram of an optoelectronic hybrid computing network is provided for an embodiment of the present application. DETAILED DESCRIPTION
[0019] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described below in connection with the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.
[0020] Currently, the computing load of the Transformer mainly consists of large-scale matrix multiplication operations, especially the query-key mapping ( ) and attention-value mapping ( ) operations in the self-attention mechanism, whose computational complexity has a quadratic relationship with the sequence length, which is particularly significant when processing long sequence tasks (such as long document understanding and high-resolution image segmentation). Performing these operations on traditional electronic chips requires frequent transfer of a large amount of weight and activation value data between the computing unit and the high-bandwidth internal, which is deeply affected by the “memory wall” problem. Not only does the data generate high latency, but its power consumption is also much higher than the computation itself, becoming the main bottleneck of improving computing power and energy efficiency.
[0021] Photon computing provides a new path for breaking through the inference bottleneck of the Transformer with its advantages of high throughput, low transmission delay and almost zero static power consumption. In particular, the optical matrix computing unit of the Mach-Zehnder interferometer (MZI, Mach-Zehnder Interferometer) can realize matrix multiplication operations through phase modulation, which can theoretically complete linear transformation in the instant of light propagation. Its main energy consumption only exists in the electro-optical conversion (E / O, Electro-Optical conversion) and opto-electronic conversion (O / E, Opto-Electronic conversion) links, which is expected to completely break through the limitations of the “memory wall” and the “power wall”, and provide exponential energy efficiency improvement for the next generation of AI (Artificial Intelligence, Artificial Intelligence) computing. Although optoelectronic computing has great potential in linear computation, it still faces many challenges in nonlinear activation, dynamic weight update and system integration. For example, optical implementation of nonlinear functions often needs to rely on electrical assistance.
[0022] Therefore, it is inevitable to develop an optoelectronic hybrid computing architecture. The architecture aims to take full advantage of the high efficiency of photon computing in linear transformation (such as matrix multiplication in the self-attention mechanism), while using electronic chips to flexibly process nonlinear operations (such as activation functions, layer normalization) and control logic. By optoelectronic heterogeneous integration technology, the MZI matrix computing unit, high-speed optoelectronic modulator and detector, and silicon-based CMOS (Complementary Metal-Oxide-Semiconductor) logic circuit are closely combined to build an on-chip optoelectronic computing system, which is expected to realize high energy efficiency and low delay inference of the Transformer model.
[0023] Based on this, the optoelectronic hybrid computing method and network provided by the embodiments of the present application can be applied to the inference of the Transformer model, can significantly improve the inference efficiency, reduce the power consumption cost, and can be widely applied to the fields of intelligent driving, industrial detection and intelligent security, and meet the low delay and high energy efficiency requirements of real-time target detection.
[0024] In order to facilitate the understanding of the present embodiment, first, a kind of optoelectronic hybrid computing method disclosed by the present embodiment is introduced in detail.
[0025] Embodiment one: The present embodiment provides an optoelectronic hybrid computing method, referring to Figure 1 The flow chart of the optoelectronic hybrid computing method, the optoelectronic hybrid computing method includes the following steps: Step S102, the electrical signal of the input data is converted into optical signal by electro-optical conversion, and the optical signal of the fixed weight matrix is pre-modulated.
[0026] Referring to Figure 2 The structure diagram of the optoelectronic hybrid computing network, in the present embodiment, system initialization can be carried out first, including: electro-optical conversion of input data and pre-modulation of fixed weight matrix.
[0027] In some embodiments, the input data can be loaded in the high-bandwidth memory, and the input data is transmitted to the static random access memory; the electrical signal of the input data is read from the static random access memory; the amplitude or phase information of the electrical signal is loaded onto the optical carrier through the digital-to-analog converter and the electro-optical modulator, forming an optical vector carrying information as the optical signal of the input data.
[0028] The input data in this embodiment is loaded from the high bandwidth memory (HBM), transmitted and cached in the static random access memory (SRAM). Then the read electrical signal is converted by the digital-to-analog converter (DAC) and the electro-optic modulator (EOM), and the amplitude or phase information of the electrical signal is loaded onto the optical carrier to form an optical signal carrying information as the optical signal of the input data.
[0029] In some embodiments, the electrical signal of the weight matrix can be pre-modulated by singular value decomposition and fixed in the optical parameters of the Mach-Zehnder interferometer network to obtain the optical signal of the weight matrix.
[0030] In this embodiment, the weight matrix that needs to be fixed (for example, the weight matrix in Self-Attention , , and the weight matrix in FFN , ) can be pre-modulated by singular value decomposition (SVD) and fixed in the optical parameters of the MZI to obtain the optical signal of the weight matrix for subsequent optical matrix multiplication operation.
[0031] In some embodiments, the Mach-Zehnder interferometer is used to construct an optical computing unit for performing linear transformation calculation in neural network forward inference; wherein the linear transformation calculation includes: matrix multiplication and linear transformation.
[0032] Referring to the structural schematic diagram of a Mach-Zehnder interferometer shown in Figure 3 In this embodiment, a Mach-Zehnder interferometer array can be used to construct a programmable optical computing unit responsible for linear transformation calculation (such as matrix multiplication and linear transformation) in neural network forward inference.
[0033] In some embodiments, the non-linear calculation and control logic in the neural network are performed through a graphics processor platform.
[0034] In this embodiment, the GPU (graphics processing unit) platform can also be used to implement the non-linear calculation and control logic in the transformer structure to achieve high-performance and low-power computing acceleration.
[0035] Step S104, performing self-attention mechanism optoelectronic hybrid calculation based on the optical signals of the input data and the optical signals of the weight matrix to obtain an optoelectronic hybrid calculation result.
[0036] As shown in Figure 2 , the self-attention mechanism optoelectronic hybrid calculation can be performed in the embodiment.
[0037] In some embodiments, the optical signals of the query vector, the key vector and the value vector can be generated based on the optical signals of the input data and the optical signals of the weight matrix; the multiplication operation of the query vector and the transposed key vector is performed through the optical matrix core to obtain a calculation result; the calculation result is converted into an electrical signal through the optoelectronic conversion module and the analog-to-digital converter, and the electrical signal of the calculation result is subjected to function operation to obtain an electrical signal of the attention weight; the electrical signal of the attention weight is converted into an optical signal through the modulator, and the attention weighted multiplication of the attention weight and the value vector is performed through the optical matrix core to obtain an optoelectronic hybrid calculation result.
[0038] As shown in Figure 2 , the self-attention mechanism optoelectronic hybrid calculation in the embodiment mainly includes the following steps 1-3: Step 1, the main controller first generates a query vector , a key vector and a value vector . The multiplication operation of and (the transpose of the key vector ) is completed by the dynamic reconstruction optical matrix core.
[0039] Among them, the optical signal of the input data can be injected into the Mach-Zehnder interferometer grid, and the matrix multiplication is performed through the Mach-Zehnder interferometer grid to output the optical signals of the query vector , the key vector and the value vector .
[0040] Step 2, after the calculation result is converted into an electrical signal through the optoelectronic conversion module and the analog-to-digital converter, the electrical signal of the attention weight is obtained by inputting the electronic calculation unit to perform function operation, so as to be prepared for calculating the attention weight.
[0041] Step 3, the obtained attention weight is converted into an optical signal through the digital-to-analog converter and the electro-optical detector, and the optical matrix multiplication of is completed by the dynamic reconstruction optical matrix core to obtain an optoelectronic hybrid calculation result.
[0042] Step S106, performing a feedforward neural network optoelectronic hybrid calculation based on the optical signal of the light-electricity hybrid calculation result and the weight matrix to obtain a feedforward neural network optoelectronic hybrid calculation result.
[0043] As shown in Figure 2 , the feedforward neural network optoelectronic hybrid calculation can be performed in the embodiment.
[0044] In some embodiments, the light-electricity hybrid calculation result can be injected into a first optical core configured with the weight matrix to perform optical matrix multiplication calculation to obtain a first linear transformation result of the optical signal; the first linear transformation result of the optical signal can be converted into an electrical signal through an analog-to-digital converter and a photoelectric detector, and the electrical signal after the first linear transformation can be subjected to a GELU function operation to obtain a GELU activated electrical signal; the GELU activated electrical signal can be converted into an optical signal through a digital-to-analog converter and an electro-optical detector; and the GELU activated optical signal can be injected into a second optical core configured with the weight matrix to perform optical matrix multiplication calculation to obtain a second linear transformation result of the optical signal, which is taken as the feedforward neural network optoelectronic hybrid calculation result.
[0045] As shown in Figure 2 , the feedforward neural network optoelectronic hybrid calculation in the embodiment mainly includes the following steps 1-3: Step 1, for the feedforward network calculation, the output optical signal x of the self-attention mechanism is directly subjected to matrix multiplication through the pre-configured weight fixed optical matrix to obtain a first linear transformation result of the optical signal.
[0046] Step 2, the first linear transformation result is converted into an electrical signal through an analog-to-digital converter and a photoelectric detector, and the electrical signal is input into an electronic calculation unit to perform a GELU function operation to obtain a GELU activated electrical signal.
[0047] Step 3, the GELU activated electrical signal is converted into an optical signal through a digital-to-analog converter and an electro-optical detector, and the optical signal is subjected to optical matrix multiplication with the optical matrix to obtain a second linear transformation result of the optical signal.
[0048] Step S108, sequentially performing residual connection calculation and layer normalization processing on the feedforward neural network optoelectronic hybrid calculation result to obtain a final output result.
[0049] In the embodiment, the optical signal after the feedforward neural network processing can be converted into an electrical signal through a photoelectric detector and an analog-to-digital converter, and the final electronic calculation operation (including residual connection calculation and layer normalization processing) is performed to obtain a final output result. As shown in Figure 2 , the residual connection calculation and layer normalization processing can include: Function operations, Operations and nonlinear activation operations.
[0050] This invention provides a hybrid optoelectronic computing method. The method involves electro-optical conversion of the input data's electrical signal to obtain an optical signal, and pre-modulating the optical signal with a fixed weight matrix. Based on the input data's optical signal and the weight matrix's optical signal, a self-attention mechanism is used for hybrid optoelectronic computing to obtain the hybrid computing result. Based on the hybrid computing result and the weight matrix's optical signal, a feedforward neural network is used for hybrid optoelectronic computing to obtain the feedforward neural network's hybrid computing result. The feedforward neural network's hybrid computing result is then subjected to residual connection calculation and layer normalization processing to obtain the final output result. This method maintains accuracy comparable to pure electronic computing in VIT while significantly improving inference efficiency and reducing power consumption costs. It can be widely applied in fields such as intelligent driving, industrial inspection, and intelligent security, meeting the low latency and high energy efficiency requirements of real-time target detection.
[0051] Example 2: This embodiment provides another optoelectronic hybrid computing method, which is implemented based on the above embodiment. Taking the forward propagation process of an encoder layer in a Transformer as an example, it focuses on describing the specific steps of the optoelectronic hybrid computing method for Transformer model inference. See Figure 4 The flowchart shown represents another optoelectronic hybrid computing method, which includes the following steps: Step S402: The electrical signal of the input data is converted into an optical signal by electro-optic conversion, and the optical signal is pre-modulated with a fixed weight matrix.
[0052] In this embodiment, photoelectric conversion and distribution of input data can be performed: the input data is first converted into a signal by a digital-to-analog converter (DAC), and then the electrical signal is converted into an optical signal by an electro-optic modulator. The converted optical signal is distributed to the corresponding optical computing core through an optical waveguide to prepare for subsequent calculations.
[0053] Step S404: Perform photoelectric hybrid calculation using a self-attention mechanism based on the optical signal of the input data and the optical signal of the weight matrix to obtain the photoelectric hybrid calculation result.
[0054] See also Figure 5 The diagram illustrates a self-attention mechanism for optoelectronic hybrid computing. Self-attention computing is a core component of the Transformer, enabling the collaborative processing of components decomposed into linear (optical) and nonlinear (electric) parts. It mainly includes the following steps: (1) Query ( ),key( ),value( Optical calculation of vectors: input electrical signal After being converted into optical signals by a DAC and an electro-optic modulator, the signals are injected into a pre-programmed MZI mesh (representing the weight matrix). , , To perform matrix multiplication, dynamically generate , , Vector-like optical signals. The process is shown in the following formula: , , ; (2) Attention score Optical calculations: and A vector of light signals is input into the core of a reconstructable optical matrix, and multiplication is performed at the speed of light using the principle of light interference. The process is shown in the following formula: .
[0055] (3) Electrical calculations of functions: The results are converted into electrical signals by a photodetector and an analog-to-digital converter, and the attention weights are calculated by the softmax function executed by the electronic computing unit. This introduces nonlinearity. The process is shown in the following formula: .
[0056] (4) Attention-weighted optical computation: Attention weights of function output Converted into an optical signal by a modulator, and... The optical signals along with the value vectors are input into the core of the optical matrix to complete the final attention-weighted multiplication calculation. The results of the photoelectric hybrid calculation were obtained. The process is shown in the following formula: .
[0057] Step S406: Perform photoelectric hybrid calculation based on the photoelectric hybrid calculation results and the optical signal of the weight matrix, and obtain the photoelectric hybrid calculation results of the feedforward neural network.
[0058] See also Figure 6 The diagram shown illustrates a feedforward neural network for optoelectronic hybrid computing. In this embodiment, the computation of the FFN layer typically consists of two linear transformations and an intermediate nonlinear activation function, involving the multiplication of two large matrices ( and... ) is executed in the optical domain to take advantage of its high parallelism and energy efficiency, rather than a nonlinear activation function ( ) and residual connection, layer normalization, etc. are completed in the electrical domain. The process is shown in the following formula: .
[0059] The feedforward neural network optoelectronic hybrid computation in this embodiment mainly includes the following steps: (1) Input modulation and first matrix multiplication (A·W1): The output (electrical signal) of the self-attention layer is converted into an optical signal through a DAC and an electro-optical modulator , and is injected into an optical matrix multiplication unit pre-configured with a fixed weight matrix to complete the first linear transformation , obtaining the first linear transformation result . The process is shown in the following formula: .
[0060] (2) Optoelectronic conversion and nonlinear activation (GELU): The result of the first optical matrix multiplication is converted into an electrical signal through an optoelectronic detector and an ADC, and then a nonlinear activation function is performed by an electronic computing unit to obtain an electrical signal after GELU activation . The process is shown in the following formula:
[0061] (3) Modulation again and second matrix multiplication (A·W2): The activated electrical signal is converted into an optical signal again through a DAC and an electro-optical modulator, and is injected into a second optical core pre-configured with a fixed weight matrix to complete the second linear transformation, obtaining the second linear transformation result . The process is shown in the following formula: .
[0062] In step S408, the feedforward neural network optoelectronic hybrid computation result is converted into an electrical signal through an optoelectronic detector and an analog-to-digital converter; the optoelectronic hybrid computation result and the electrical signal of the feedforward neural network optoelectronic hybrid computation result are added through a residual connection to obtain a residual connection computation result; and the residual connection computation result is subjected to layer normalization processing to obtain a final output result.
[0063] In this embodiment, residual connection computation can be performed in the data stage: the result of the second optical matrix multiplication is converted back into an electrical signal through optoelectronic conversion again, and in the electronic computing unit, the original input and the output of the FFN layer are added through a residual connection to obtain a residual connection result , and the process is shown in the following formula: .
[0064] In this embodiment, layer normalization can also be performed after the data phase: for the output of the residual connection Application layer normalization yields the final output. The process is shown in the following formula: .
[0065] In summary, this invention provides a hybrid optoelectronic computing method for Transformer model inference. A Mach-Zehnder interferometer (MZI) array is used to construct a programmable optical computing unit responsible for linear transformation calculations (such as matrix multiplication and linear transformations) in the forward inference of the neural network. Simultaneously, a GPU platform is utilized to implement nonlinear calculations and control logic in the Transformer structure, achieving high-performance and low-power computational acceleration. A high-speed optoelectronic conversion module enables efficient conversion between optical and electrical signals, and a unified task scheduler is established to achieve pipelined parallelism of optical and electronic computing. This method maintains accuracy comparable to pure electronic computing in Visual Transformer (VIT) while significantly improving inference efficiency and reducing power consumption costs. It can be widely applied in fields such as intelligent driving, industrial inspection, and intelligent security, meeting the low latency and high energy efficiency requirements of real-time target detection.
[0066] The method provided in this embodiment of the invention employs a photoelectric hybrid computing architecture to implement transformer inference. First, input data... After the system initialization module performs electro-optic conversion of the input data and pre-modulates a fixed weight matrix. The input data is then processed through a self-attention mechanism opto-optical hybrid computing module for corresponding optical and electrical calculations, followed by a feedforward neural network opto-optical hybrid computing module for the same calculations. Both modules involve a light-to-electricity-to-light process. Finally, the output module performs subsequent electronic domain processing. In practice, during system initialization, pre-configured fixed weights (such as FFN layer weights) are pre-configured to the optical core. During forward inference, the input data is converted into optical signals via electro-optic modulation and input to the corresponding optical matrix core for multiplication and addition operations. The results are then fed back to the electronic domain via the opto-optical conversion module for further processing. For dynamically generated weights in the self-attention mechanism, the system calculates and configures them in real-time to the dynamic optical core using the electronic computing unit, enabling light-speed matrix multiplication.
[0067] Example 3: Corresponding to the above method embodiments, this invention provides a photoelectric hybrid computing network, see [link to relevant documentation]. Figure 7 The diagram shows a structural schematic of a hybrid optoelectronic computing network, which includes: The system initialization module 71 is used to perform electro-optic conversion on the electrical signal of the input data to obtain an optical signal, and to pre-modulate the optical signal with a fixed weight matrix; The self-attention mechanism optoelectronic hybrid computing module 72 is used to perform self-attention mechanism optoelectronic hybrid computing based on the optical signal of the input data and the optical signal of the weight matrix to obtain the optoelectronic hybrid computing result. The feedforward neural network optoelectronic hybrid calculation module 73 is used to perform feedforward neural network optoelectronic hybrid calculation based on the optoelectronic hybrid calculation results and the optical signal of the weight matrix, and obtain the feedforward neural network optoelectronic hybrid calculation results. The final output module 74 is used to perform residual connection calculation and layer normalization processing on the optoelectronic hybrid calculation results of the feedforward neural network in sequence to obtain the final output result.
[0068] This invention provides a hybrid optoelectronic computing network. It performs electro-optical conversion on the input data's electrical signal to obtain an optical signal, and pre-modulates the optical signal with a fixed weight matrix. Based on the input data's optical signal and the weight matrix's optical signal, it performs a self-attention mechanism for hybrid optoelectronic computing to obtain a hybrid optoelectronic computing result. Based on the hybrid optoelectronic computing result and the weight matrix's optical signal, it performs a feedforward neural network for hybrid optoelectronic computing to obtain a feedforward neural network for hybrid optoelectronic computing result. The feedforward neural network's hybrid optoelectronic computing result is then subjected to residual connection calculation and layer normalization processing to obtain the final output result. This method maintains accuracy comparable to pure electronic computing in VIT while significantly improving inference efficiency and reducing power consumption costs. It can be widely applied in fields such as intelligent driving, industrial inspection, and intelligent security, meeting the low latency and high energy efficiency requirements of real-time target detection.
[0069] The aforementioned system initialization module is used to load input data into high-bandwidth memory, transmit the input data to static random access memory, read the electrical signal of the input data from static random access memory, and load the amplitude or phase information of the electrical signal onto the optical carrier through a digital-to-analog converter and an electro-optic modulator to form an optical vector carrying information as the optical signal of the input data.
[0070] The aforementioned system initialization module is used to pre-modulate and fix the electrical signal of the weight matrix into the optical parameters of the Mach-Zehnder interferometer network through singular value decomposition, thereby obtaining the optical signal of the weight matrix.
[0071] The Mach-Zehnder interferometer is used to construct the optical computing unit, which is used to perform linear transformation calculations in the forward inference of the neural network; the linear transformation calculations include matrix multiplication and linear transformation.
[0072] The aforementioned optoelectronic hybrid computing network performs nonlinear calculations and control logic in the neural network through a graphics processing unit platform.
[0073] The aforementioned self-attention mechanism optoelectronic hybrid computing module generates optical signals for query vectors, key vectors, and value vectors based on the optical signals of the input data and the weight matrix; it obtains the calculation result by multiplying the query vector with the transposed key vector through the optical matrix core; and it converts the calculation result into an electrical signal through an optoelectronic conversion module and an analog-to-digital converter, and then processes the electrical signal of the calculation result. The electrical signal of the attention weight is obtained by function operation; the electrical signal of the attention weight is converted into an optical signal by modulator; and the attention weight and value vector are calculated by attention weight multiplication through the core of the optical matrix to obtain the optoelectronic hybrid calculation result.
[0074] The aforementioned self-attention mechanism optoelectronic hybrid computing module is used to inject the optical signal of the input data into the Mach-Zehnder interferometer grid, perform matrix multiplication through the Mach-Zehnder interferometer grid, and output the optical signals of the query vector, key vector, and value vector.
[0075] The aforementioned feedforward neural network optoelectronic hybrid computing module is used to inject the optoelectronic hybrid computing result into the first optical core of the configuration weight matrix to perform optical matrix multiplication calculation to obtain the optical signal of the first linear transformation result; convert the optical signal of the first linear transformation into an electrical signal through an analog-to-digital converter and a photodetector; perform GELU function operation on the electrical signal of the first linear transformation to obtain the GELU-activated electrical signal; convert the GELU-activated electrical signal into an optical signal through a digital-to-analog converter and an electro-optical detector; inject the GELU-activated optical signal into the second optical core of the configuration weight matrix to perform optical matrix multiplication calculation to obtain the optical signal of the second linear transformation result; and use the optical signal of the second linear transformation result as the optoelectronic hybrid computing result of the feedforward neural network.
[0076] The aforementioned final output module is used to convert the optoelectronic hybrid calculation results of the feedforward neural network into electrical signals through a photodetector and an analog-to-digital converter; add the electrical signals of the optoelectronic hybrid calculation results and the feedforward neural network optoelectronic hybrid calculation results through residual connection to obtain the residual connection calculation results; and perform layer normalization processing on the residual connection calculation results to obtain the final output results.
[0077] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described optoelectronic hybrid computing network can be referred to the corresponding process in the aforementioned embodiments of the optoelectronic hybrid computing method, and will not be repeated here.
[0078] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.
[0079] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the referred element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0080] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A photoelectric hybrid computing method, characterized in that, The method includes: The electrical signal of the input data is converted into an optical signal by electro-optic conversion, and the optical signal is pre-modulated with a fixed weight matrix. Based on the optical signal of the input data and the optical signal of the weight matrix, a self-attention mechanism photoelectric hybrid calculation is performed to obtain the photoelectric hybrid calculation result. Based on the optoelectronic hybrid calculation results and the optical signal of the weight matrix, a feedforward neural network optoelectronic hybrid calculation is performed to obtain the feedforward neural network optoelectronic hybrid calculation results. The feedforward neural network optoelectronic hybrid calculation results are sequentially subjected to residual connection calculation and layer normalization processing to obtain the final output result.
2. The method according to claim 1, characterized in that, The steps of electro-optical conversion of input data electrical signals to obtain optical signals include: Load the input data into high-bandwidth memory and transfer the input data to static random access memory; An electrical signal that reads the input data from the static random access memory; The amplitude or phase information of the electrical signal is loaded onto the optical carrier by a digital-to-analog converter and an electro-optic modulator to form an optical vector carrying information as the optical signal of the input data.
3. The method according to claim 1, characterized in that, The steps of premodulating an optical signal with a fixed weight matrix include: The electrical signal of the weight matrix is pre-modulated and fixed in the optical parameters of the Mach-Zehnder interferometer network through singular value decomposition to obtain the optical signal of the weight matrix.
4. The method according to claim 3, characterized in that, The Mach-Zehnder interferometer is used to construct an optical computing unit, which is used to perform linear transformation calculations in the forward inference of a neural network; wherein, the linear transformation calculations include matrix multiplication and linear transformation.
5. The method according to claim 4, characterized in that, Nonlinear computation and control logic in neural networks are performed using a graphics processing unit (GPU) platform.
6. The method according to claim 3, characterized in that, The steps for performing self-attention mechanism photoelectric hybrid calculations based on the optical signals of the input data and the optical signals of the weight matrix to obtain the photoelectric hybrid calculation results include: Based on the optical signals of the input data and the optical signals of the weight matrix, optical signals for generating query vectors, key vectors, and value vectors are generated. The calculation result is obtained by multiplying the query vector and the transposed key vector through the optical matrix core. The calculation results are converted into electrical signals using a photoelectric conversion module and an analog-to-digital converter, and the electrical signals of the calculation results are then processed. Function operations yield electrical signals for attention weights; The electrical signal of the attention weight is converted into an optical signal by a modulator, and the attention weight and the value vector are multiplied by the attention weight through the core of the optical matrix to obtain the optoelectronic hybrid calculation result.
7. The method according to claim 6, characterized in that, The steps of generating optical signals for query vectors, key vectors, and value vectors based on the optical signals of the input data and the optical signals of the weight matrix include: The optical signal of the input data is injected into the Mach-Zehnder interferometer grid, and matrix multiplication is performed through the Mach-Zehnder interferometer grid to output the optical signals of the query vector, key vector, and value vector.
8. The method according to claim 1, characterized in that, The steps for performing feedforward neural network photoelectric hybrid calculations based on the photoelectric hybrid calculation results and the optical signals of the weight matrix to obtain the feedforward neural network photoelectric hybrid calculation results include: The optoelectronic hybrid calculation result is injected into the first optical core configured with the weight matrix to perform optical matrix multiplication calculation to obtain the optical signal of the first linear transformation result; The optical signal of the first linear transformation is converted into an electrical signal through an analog-to-digital converter and a photodetector, and the electrical signal of the first linear transformation is subjected to the GELU function operation to obtain the GELU-activated electrical signal. The electrical signal activated by the GELU is converted into an optical signal using a digital-to-analog converter and an electro-optic detector. The optical signal activated by the GELU is injected into the second optical core configured with the weight matrix to perform optical matrix multiplication calculation to obtain the optical signal of the second linear transformation result. The optical signal of the second linear transformation result is used as the optoelectronic hybrid calculation result of the feedforward neural network.
9. The method according to claim 1, characterized in that, The steps for performing residual connection calculations and layer normalization on the feedforward neural network optoelectronic hybrid calculation results to obtain the final output result include: The photoelectric hybrid calculation results of the feedforward neural network are converted into electrical signals using a photodetector and an analog-to-digital converter. The electrical signal of the optoelectronic hybrid calculation result and the optoelectronic hybrid calculation result of the feedforward neural network are added together through residual connection to obtain the residual connection calculation result; The residual connection calculation results are subjected to layer normalization to obtain the final output results.
10. A hybrid optoelectronic computing network, characterized in that, The optoelectronic hybrid computing network includes: The system initialization module is used to perform electro-optic conversion on the electrical signal of the input data to obtain an optical signal, and to pre-modulate the optical signal with a fixed weight matrix; The self-attention mechanism optoelectronic hybrid computing module is used to perform self-attention mechanism optoelectronic hybrid computing based on the optical signal of the input data and the optical signal of the weight matrix to obtain the optoelectronic hybrid computing result. The feedforward neural network optoelectronic hybrid computing module is used to perform feedforward neural network optoelectronic hybrid computing based on the optoelectronic hybrid computing results and the optical signal of the weight matrix, so as to obtain the feedforward neural network optoelectronic hybrid computing results. The final output module is used to perform residual connection calculation and layer normalization processing on the optoelectronic hybrid calculation results of the feedforward neural network in sequence to obtain the final output result.