Large model computing device and method based on MZI cascade network

Abstract

The invention discloses a large model calculation device and method based on an MZI cascade network, and relates to the technical field of model calculation. According to the method, a photon artificial intelligence chip and an FPGA technology are combined, the problem that an existing Transform model is high in energy consumption during operation is solved, the system comprises three main components including an FPGA control module, an optical calculation module and an FPGA calculation module, the FPGA control module is responsible for digital-to-analog conversion and time sequence input of signals, and the optical calculation module is responsible for digital-to-analog conversion of the signals and time sequence input of the signals. The optical calculation module is used for weight calculation and feature extraction of data, the FPGA calculation module is responsible for all calculation tasks except the optical calculation module, and the main purpose of the invention is to introduce optical calculation into calculation of a Transformer model, realize matrix operation by means of a photon artificial intelligence chip, and improve the calculation efficiency. Therefore, the high performance is maintained, the power consumption during model operation is reduced, the calculation flexibility of the FPGA and the characteristics of high speed and high efficiency of optical calculation are combined, and photoelectric hybrid calculation becomes a powerful solution for realizing low-energy-consumption Transform model calculation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an optical Transformer implementation method based on an MZI cascade network. Background Art

[0002] Transformer is a neural network architecture that has achieved great success in natural language processing (NLP) and other sequence data processing tasks. Compared with traditional recurrent neural networks (RNN) and convolutional neural networks (CNN), Transformer has many unique features and advantages. First, it uses a self-attention mechanism that can simultaneously consider the information of all positions in the input sequence, thereby better capturing long-distance dependencies. Second, the parallelization of the self-attention mechanism allows Transformer to process sequence data efficiently without being troubled by the problem of gradient vanishing or gradient exploding. These advantages make it perform well in tasks such as machine translation and text generation. At the same time, Transformer is not only used for NLP tasks, but can also be applied to computer vision, audio processing and other fields. This versatility makes it also potential in multimodal tasks.

[0003] Before the emergence of Transformer, Seq2Seq model was the mainstream method for processing sequence data. This model usually uses recurrent neural network (RNN) or long short-term memory (LSTM) network. The foundation of Transformer technology is the paper "Attention is All You Need" proposed by Vaswani et al. of the Google Brain team in 2017. This paper introduced the concept of self-attention mechanism, which can effectively handle the relationship in the input sequence, thereby achieving better performance in tasks such as machine translation. In 2018, Google released the BERT model, which is a pre-trained Transformer model. BERT improves the performance of natural language understanding tasks and leads the research trend of transfer learning using pre-trained models. The success of BERT has inspired the exploration of Transformer technology in various fields. In 2020 and beyond, Transformer technology began to be applied to image processing, speech recognition and other non-language tasks, showing its wide applicability. Transformer technology is still evolving, and more innovations may appear in the future, including more effective model architectures, applications in more fields, and exploration of model interpretability and robustness.

[0004] However, Transformer currently has some limitations. First, the training process of the Transformer model requires a large amount of data and high-performance computing resources. These models usually rely on thousands of GPU or TPU cores and run continuously for weeks or even months. Such huge computing requirements lead to huge power consumption. As the scale of the model continues to expand, this energy consumption problem becomes more and more serious. Secondly, model training increases the cost of research and development. This means that only large companies or research institutions with sufficient funds have the ability to develop and maintain such models, which to a certain extent limits the popularization and innovation of technology. In addition, once the models are trained, they also require relatively high energy consumption when performing specific tasks. Although this runtime energy consumption is usually much lower than the training stage, when these models are widely used in various online services and applications, the accumulated energy consumption is still a problem that cannot be ignored. Traditional single data processing calculations cannot map all input signals from the real number domain to the positive number domain that can be represented by light intensity, which increases the calculation time and increases the power consumption of the model during operation.

[0005] The emergence of photonic AI chips can effectively solve this problem. It uses optics rather than traditional electronics for data processing and calculation. This method provides unique advantages in deep learning, especially when dealing with complex models such as Transformer. Photonic AI chips can significantly reduce energy consumption when processing data. This is because during optical computing, information is transmitted quickly and does not generate much heat. In contrast, traditional electronic-based chips generate a lot of heat when processing large amounts of data, which not only increases the demand for cooling systems, but also significantly increases energy consumption. Therefore, applying photonic technology to AI chips, especially when running large models such as Transformer, can effectively reduce energy consumption and reduce dependence on cooling systems. Summary of the invention

[0006] The present application provides a large model calculation method, device and equipment based on an MZI cascade network to solve the problem of excessive energy consumption during the operation of Transformer large model calculations, introduce optical calculations into the calculation of the model, and implement matrix operations through photonic artificial intelligence chips, thereby reducing the power consumption of the model during operation.

[0007] In a first aspect, the present application provides a large model calculation method based on an MZI cascade network, the method comprising the following steps:

[0008] Step 1: The input data is converted from digital signal to analog signal through the DA module, and the converted analog signal is input into multiple parallel optical linear calculation modules by the FPGA control module;

[0009] Step 2: After receiving the input, multiple light linear calculation modules perform weight calculation, with light intensity as output. The light intensity signal is converted from an analog signal to a digital signal through the AD module, and the FPGA control module sends the digital signal to the FPGA calculation module for self-attention calculation;

[0010] Step 3: The self-attention output calculated by the FPGA calculation module is converted into an analog signal by the DA module;

[0011] Step 4: The analog signal converted by the DA module in step 3 is input into the light linearity calculation module under the control of the FPGA control module, and the light intensity output is obtained through calculation;

[0012] The light intensity output described in step 5 and step 4 is converted into a digital signal output through an AD module.

[0013] In one possible implementation, the optical linear calculation module is divided into transforming the input signal, and the transformed signal is output by the photonic artificial intelligence chip and then output after undergoing an inverse transformation relative to the aforementioned transformation, mapping all input signals from the real number domain to the positive number domain that can be represented by light intensity.

[0014] In a possible implementation, the formula for calculating the self-attention calculation in step 2 is:

[0015] out=QK T

[0016] out'=softmax(out)

[0017] O=Vout'.

[0018] In a second aspect, the present application provides a large model computing device based on an MZI cascade network, comprising:

[0019] The FPGA control module is used to perform digital-to-analog conversion on the part that needs signal conversion in the whole process, and input the timing of signal setting into the optical computing module, and the FPGA control module is connected to the input signal;

[0020] The optical computing module is used to perform weight calculation on the data and obtain the features of the data to input into the FPGA computing module;

[0021] The FPGA calculation module is used for calculations other than the tasks of the optical calculation module in the whole process. The FPGA calculation module is connected to the FPGA control module and the optical calculation module, and the FPGA calculation module outputs the calculation results.

[0022] In a possible implementation, the FPGA control module includes a clock module, an AD / DA module, an input / output control system, and an FPGA chip;

[0023] The FPGA chip is connected to the AD / DA module, the input / output control system, and the clock module to provide computing power;

[0024] The AD / DA module is connected to the input and output control system to provide mutual conversion between digital signals and analog signals;

[0025] The clock module is connected to the FPGA chip to provide a timing signal;

[0026] The input-output control system is connected to the FPGA chip and the AD / DA module, and is used to perform communication control of input and output signals on the FPGA chip.

[0027] In a possible implementation, the optical computing module includes at least four photon linear computing modules, lasers, and detectors. The computing process is that the signal converted by the DA module of the FPGA control module passes through three of the photon linear computing modules, and the output is then converted by the AD module of the FPGA control module and input into the FPGA computing module for self-attention calculation. The calculation result is converted by the DA module controlled by the FPGA and input into the fourth photon linear computing module, and the output is then converted by the AD module of the FPGA control module and input into the FPGA computing module. Among them, the photon linear transformation computing module includes the transformation calculation and inverse transformation calculation provided by the FPGA computing module, and the photon artificial intelligence chip.

[0028] In one possible implementation, the laser is connected to the photonic artificial intelligence chip to provide continuous light source input to the chip. The FPGA control module is connected to the photonic artificial intelligence chip. The photonic artificial intelligence chip modulates the input of the laser according to the data sent by the FPGA control module. The photonic artificial intelligence chip is composed of a cascade array of Mach-Zehnder interferometers, and the modulation process is as follows:

[0029] Step 1: Adjust the modulator according to the voltage signal sent by the FPGA control module. The modulator modulates the input signal into the pulse signal input by the laser;

[0030] Step 2: According to the neural network weight information sent by the FPGA control module, the modulator in the Mach-Zehnder interferometer cascade array in each photonic artificial intelligence chip is adjusted by voltage to modulate the weight information onto the optical pulse;

[0031] Step 3: In the optical domain, a multiplication operation is performed on the input signal modulated on the optical pulse and the weight signal.

[0032] The large model calculation method, device and equipment based on the MZI cascade network provided in this application introduces optical computing into the calculation of the model, realizes matrix operations through photonic artificial intelligence chips, thereby reducing the power consumption of the model during operation. Optical computing is introduced into the calculation of the Transformer model, and matrix operations are realized with the help of photonic artificial intelligence chips, thereby reducing the power consumption of the model during operation while maintaining high performance. This architecture not only solves the energy consumption problem, but also combines the computing flexibility of FPGA and the high speed and high efficiency of optical computing, making optoelectronic hybrid computing a powerful solution for realizing low-energy Transformer model calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0034] Figure 1 The main components and connection methods of the optical Transformer neural network system architecture provided for this application.

[0035] Figure 2 The main computing part of the optical Transformer neural network system architecture provided in this application.

[0036] Figure 3 The composition of the photon linear calculation module provided in this application.

[0037] Figure 4 A diagram of the process of modulating input and weight information onto light provided for this application.

[0038] Figure 5 Diagram of the FPGA control module and FPGA calculation module provided for this application.

[0039] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0040] The technical scheme and hardware structure in the embodiments of the present invention will be fully described below in conjunction with the drawings in the embodiments of the present invention. Unless otherwise specifically stated, the relative arrangement of the components and steps described in these embodiments does not limit the scope of the present invention. In order to make the objects, technical schemes and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail below with reference to the drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without paying creative work should fall within the protection scope of the present invention.

[0041] Based on the photonic artificial intelligence chip, the present invention proposes an optical Transformer neural network system architecture, which can reduce the energy consumption during the operation of the Transformer model and significantly reduce the training and operation costs of large models based on the Transformer.

[0042] See attached Figure 1 ,This system includes FPGA control module, optical computing module, and FPGA computing module.,The three modules cooperate with each other for processing.,The input must first be controlled by the FPGA control module, and then input to the optical computing module for calculation.,In this process, the FPGA module performs necessary nonlinear and linear operations.

[0043] See attached Figure 2 The key computing parts of the Transformer model include DA / AD module, optical linear computing module, and FPGA computing module. The main steps of its implementation are:

[0044] Step 1: The input data is converted from digital signals to analog signals through the DA module, and these converted analog signals are input into three parallel optical linear calculation modules respectively by the FPGA control module.

[0045] Step 2: After receiving the input, the three light linear calculation modules perform weight calculation and take light intensity as output. These light intensity signals are converted from analog signals to digital signals through the AD module. The FPGA control module sends these digital signals to the FPGA calculation module for self-attention calculation.

[0046] Step 3: The self-attention output calculated by the FPGA computing module is converted into an analog signal by the DA module.

[0047] Step 4: The analog signal converted by the DA module in step 3 is input into the fourth optical linear calculation module under the control of the FPGA control module to obtain the light intensity output through calculation.

[0048] Step 5: The light intensity output described in step 4 is converted into a digital signal output through an AD module.

[0049] The above steps are the key calculation steps to realize the Transformer model. In this process, the calculation of the light linear calculation module refers to the attached Figure 3 , which can be divided into transforming the input signal, and then outputting the transformed signal through the photon artificial intelligence chip and then performing the inverse transformation relative to the aforementioned transformation. Since the input of the photon artificial intelligence chip is the light intensity value, and the input signal may include negative values, this transformation allows all input signals to be mapped from the real number domain to the positive number domain that can be represented by the light intensity.

[0050] See attached Figure 4 The laser is connected to the photon artificial intelligence chip to provide continuous light source input to the chip. The FPGA control module is connected to the photon artificial intelligence chip, and the photon artificial intelligence chip modulates the laser input according to the data sent by the FPGA control module. The photon artificial intelligence chip is composed of a cascade array of Mach-Zehnder interferometers, and its modulation process is:

[0051] Step 1: Adjust the modulator according to the voltage signal sent by the FPGA control module. The modulator modulates the input signal into the pulse signal input by the laser.

[0052] Step 2: According to the neural network weight information sent by the FPGA control module, the modulator in the Mach-Zehnder interferometer cascade array in each photonic artificial intelligence chip is adjusted by voltage to modulate the weight information onto the optical pulse.

[0053] Step 3: In the optical domain, a multiplication operation is performed on the input signal modulated on the optical pulse and the weight signal.

[0054] Using FPGAs for optical-electrical hybrid computing has several advantages, especially when dealing with high-performance computing tasks such as deep learning and big data analysis. FPGAs combine the flexibility of traditional electronic computing with the high speed and efficiency of optical computing, making optical-electrical hybrid computing a promising research and application field.

[0055] The structure of the FPGA calculation and control module can refer to the attached Figure 5 The FPGA control module is used to realize real-time high-speed communication between the optical computing module and the multi-task neural network module, and to realize real-time update of the optical computing module weight, and includes: FPGA chip, AD / DA module, input and output control system, and clock module. The functions of the FPGA computing module include: providing basic multiplication, summation, nonlinear calculation, and linear transformation calculation, which are the computing functions provided by the FPGA chip.

[0056] The FPGA chip is connected to the AD / DA module, the input / output control system, and the clock module to provide computing power;

[0057] The AD / DA module is connected to the input and output control system to provide mutual conversion between digital signals and analog signals;

[0058] The clock module is connected to the FPGA chip to provide a timing signal;

[0059] The input-output control system is connected to the FPGA chip and the AD / DA module, and is used to perform communication control of input and output signals on the FPGA chip.

[0060] Based on the above description, the main purpose of the present invention is to introduce optical computing into the calculation of the Transformer model, and realize matrix operations with the help of photon artificial intelligence chips, so as to reduce the power consumption of the model while maintaining high performance. The system used includes three components: FPGA control module, optical computing module and FPGA computing module. The FPGA control module is responsible for the digital-to-analog conversion and timing input of the signal, while the optical computing module is used for the weight calculation and feature extraction of the data. The FPGA computing module is responsible for all computing tasks except the optical computing module. This method combines photon artificial intelligence chips and FPGA technology, and combines the computational flexibility of FPGA with the high speed and high efficiency of optical computing, so that optoelectronic hybrid computing becomes a powerful solution for realizing low-energy Transformer model calculation. This application introduces optical computing into the calculation of the model, realizes matrix operations through photon artificial intelligence chips, and solves the problem of excessive energy consumption during the calculation and operation of the Transformer large model through the method of the present invention, thereby reducing the power consumption of the model during operation, achieving the purpose of the present invention, and achieving the ideal technical effect.

Claims

1. A large model computing device based on an MZI cascade network, comprising: The FPGA control module is used to perform digital-to-analog conversion on the part that needs signal conversion in the whole process, and input the timing of signal setting into the optical computing module, and the FPGA control module is connected to the input signal; The optical computing module is used to perform weight calculation on the data and obtain the features of the data to input into the FPGA computing module; The FPGA calculation module is used for calculations other than the tasks of the optical calculation module in the whole process. The FPGA calculation module is connected to the FPGA control module and the optical calculation module, and the FPGA calculation module outputs the calculation results.

2. A large model calculation method based on an MZI cascade network, implemented by the device of claim 1, characterized in that: The method comprises the following steps: Step 1: The input data is converted from digital signal to analog signal through the DA module, and the converted analog signal is input into multiple parallel optical linear calculation modules by the FPGA control module; Step 2: After receiving the input, multiple light linear calculation modules perform weight calculation, with light intensity as output. The light intensity signal is converted from an analog signal to a digital signal through the AD module, and the FPGA control module sends the digital signal to the FPGA calculation module for self-attention calculation; Step 3: The self-attention output calculated by the FPGA calculation module is converted into an analog signal by the DA module; Step 4: The analog signal converted by the DA module in step 3 is input into the light linearity calculation module under the control of the FPGA control module, and the light intensity output is obtained through calculation; The light intensity output described in step 5 and step 4 is converted into a digital signal output through an AD module.

3. The large model calculation method based on the MZI cascade network according to claim 2 is characterized in that: The FPGA control module includes a clock system, an AD / DA conversion system, an input and output control system, and an FPGA chip.

4. The large model calculation method based on the MZI cascade network according to claim 2 is characterized in that: The optical computing module includes at least 4 photon linear computing modules, a laser, and a detector. The computing process is that the signal converted by the DA module of the FPGA control module passes through 3 of the photon linear computing modules, and the output is then converted by the AD module of the FPGA control module and input into the FPGA computing module for self-attention calculation. The calculation result is converted by the DA module controlled by the FPGA and input into the 4th photon linear computing module, and the output is then converted by the AD module of the FPGA control module and input into the FPGA computing module.

5. The large model calculation method based on the MZI cascade network according to claim 4 is characterized in that: The photon linear calculation module includes transformation calculation and inverse transformation calculation provided by the FPGA calculation module, and a photon artificial intelligence chip. The laser is connected to the photon artificial intelligence chip to provide continuous light source input for the chip. The FPGA control module is connected to the photon artificial intelligence chip, and the photon artificial intelligence chip modulates the input of the laser according to the data sent by the FPGA control module.

6. The large model calculation method based on the MZI cascade network according to claim 5 is characterized in that: The photonic artificial intelligence chip is composed of a cascade array of Mach-Zehnder interferometers.

7. The large model calculation method based on the MZI cascade network according to claim 2 is characterized in that: The formula for calculating the self-attention calculation in step 2 is: out=QK T out'=softmax(out) O=Vout'.

8. The large model calculation method based on the MZI cascade network according to claim 2 is characterized in that: The FPGA calculation module is used to provide calculation functions for the FPGA chip, including: providing basic multiplication, summation, nonlinear calculation, and linear transformation calculation.

9. The large model calculation method based on the MZI cascade network according to claim 5, characterized in that: The modulation process is: Step 1: According to the voltage signal sent by the FPGA control module, the modulator is adjusted, and the modulator modulates the input signal into the pulse signal input by the laser; Step 2: According to the neural network weight information sent by the FPGA control module, the modulator in the Mach-Zehnder interferometer cascade array in each photonic artificial intelligence chip is adjusted by voltage to modulate the weight information onto the optical pulse; Step 3: In the optical domain, a multiplication operation is performed on the input signal modulated on the optical pulse and the weight signal.

Images