Data processing method, device, storage medium and computer equipment

Abstract

The present application provides a data processing method, apparatus, storage medium, and computer equipment, the method comprising: traversing the data to be processed according to the rules of convolution calculation, encoding the initial digital domain signal corresponding to the sub-data currently traversed into a first optical domain signal; inputting the first optical domain signal into an optical neural network for convolution processing and nonlinear processing to obtain a second optical domain signal of the sub-data; the convolution kernel weights used in the convolution processing are characterized by the optical constant values ​​of the phase change materials in each storage and computing integrated unit set in the optical neural network; converting the second optical domain signals corresponding to each sub-data obtained through traversal into target digital domain signals and inputting them into a classification neural network to obtain a classification result. The present application breaks through the limitations of existing optical neural networks in terms of power consumption and depth, and has the advantages of high efficiency, low latency, and scalability, and is suitable for edge intelligent computing scenarios for tasks such as image classification.

Description

Technical Field

[0001] The present application relates to the field of optical computing, and in particular to a data processing method, apparatus, storage medium, and computer equipment. Background Art

[0002] Edge computing, a distributed computing framework initiated at the edge of the network, aims to generate faster network service responses and reduce data transmission latency. However, because edge computing typically requires limited computing power on edge devices, it can lead to high latency in neural network model inference tasks.

[0003] With the development of optical computing, replacing electrical computing with optical computing can reduce response latency and increase throughput by leveraging its unique characteristics of large bandwidth and high parallelism. Therefore, if optical computing can be applied to the neural network model used in edge computing, it is expected to reduce edge computing latency and resource consumption.

[0004] However, when optical neural network computing chips are currently used for edge computing, electricity is still used to control network weights. The large number of weights still causes optical neural networks to have problems with inference delays and high computing power consumption, which has obvious disadvantages. Summary of the Invention

[0005] In view of this, the present application provides a data processing method, apparatus, storage medium and computer equipment to reduce the inference delay and computing power consumption when optical neural networks are applied in edge computing.

[0006] Specifically, this application is implemented through the following technical solutions:

[0007] In a first aspect, an embodiment of the present disclosure provides a data processing method, including:

[0008] Traversing the data to be processed according to the convolution calculation rule, and encoding the initial digital domain signal corresponding to the currently traversed sub-data into a first optical domain signal;

[0009] Inputting the first optical domain signal into an optical neural network for at least one convolution process and nonlinear processing to obtain a second optical domain signal of the sub-data on an output channel; a convolution kernel weight used in the convolution process is characterized by an optical constant value of a phase change material in each integrated storage and computing unit provided in the optical neural network;

[0010] The second optical domain signals corresponding to the respective sub-data obtained through the traversal are converted into target digital domain signals and input into the classification neural network to obtain the classification result of the data to be processed.

[0011] In one possible implementation, the optical neural network includes at least one hidden layer, the hidden layer including a cross-waveguide array for convolution processing and a micro-ring modulation array for nonlinear processing; the cross-waveguide array includes a plurality of integrated storage and computing units;

[0012] Inputting the first light domain signal into an optical neural network for at least one convolution process and nonlinear process to obtain a second light domain signal of the sub-data on an output channel includes:

[0013] Inputting the first light domain signal into a first hidden layer in the optical neural network, and performing convolution processing on the first light domain signal using each storage-computation integrated unit in the first hidden layer to obtain a convolution result;

[0014] Inputting the convolution result into the micro-ring modulation array of the first hidden layer for nonlinear processing to obtain a nonlinear result;

[0015] In the case where a next hidden layer exists, inputting the nonlinear result into the next hidden layer to obtain a nonlinear result output by the next hidden layer;

[0016] The nonlinear result output by the last hidden layer is used as the second optical domain signal of the sub-data on the output channel.

[0017] In one possible implementation, the storage-computing integrated unit includes at least input and output waveguides, a directional coupler, and a phase change material;

[0018] The multiple storage-computing integrated units are arranged according to a set structure, each storage-computing integrated unit in a row or a column of the arrangement corresponds to a convolution kernel, and each convolution kernel weight of the convolution kernel is characterized by an optical constant value of a phase change material of each storage-computing integrated unit;

[0019] The step of performing convolution processing on the first optical domain signal using each storage-computation integrated unit in the first hidden layer to obtain a convolution result includes:

[0020] For any convolution kernel of the first hidden layer, input the first optical domain signal to each storage-computing integrated unit corresponding to the convolution kernel through each input waveguide corresponding to the convolution kernel;

[0021] For any integrated storage and computing unit, the optical energy of the first optical domain signal at a set ratio is coupled using a directional coupler in the integrated storage and computing unit, and a convolution kernel weight represented by an optical constant value of the phase change material is used for multiplication processing, and the multiplication result of the integrated storage and computing unit is obtained and output through an output waveguide;

[0022] Determining an initial convolution result corresponding to the first optical domain signal under the convolution kernel according to a multiplication result of output waveguides of the respective storage-computation integrated units corresponding to the convolution kernel;

[0023] The convolution result of the first light domain signal is determined according to the initial convolution results corresponding to the first light domain signal under each of the convolution kernels.

[0024] In a possible implementation, the micro-ring modulation array includes a plurality of micro-ring modulators, wherein a doped region is provided in the micro-ring modulator, and the doped region includes P-type semiconductor carriers and N-type semiconductor carriers;

[0025] Inputting the convolution result into the micro-ring modulation array of the first hidden layer for nonlinear processing to obtain a nonlinear result includes:

[0026] Converting the convolution result into a current signal and inputting the current signal into each micro-ring modulator in the micro-ring modulation array of the first hidden layer;

[0027] Using the current signal to drive the carrier concentration in the doped region to change, thereby changing the refractive index of the micro-ring modulator, and performing nonlinear modulation based on the changed refractive index to obtain an initial modulation result;

[0028] The nonlinear result is obtained based on the initial modulation results corresponding to each micro-ring modulator.

[0029] In a possible implementation, the micro-ring modulation array further includes a first preset modulator cascaded with each micro-ring modulator;

[0030] The performing nonlinear modulation based on the changed refractive index to obtain an initial modulation result includes:

[0031] Inputting the current signal into the first preset modulator of each micro-ring modulator cascade, and adjusting the phase of the first preset modulator;

[0032] Using an independent laser source provided for each first preset modulator, a preset optical domain signal is inputted into each first preset modulator respectively;

[0033] The preset optical domain signal is nonlinearly modulated using the adjusted phase and the changed refractive index to obtain the initial modulation result.

[0034] In a possible implementation manner, encoding the initial digital domain signal corresponding to the sub-data into a first optical domain signal includes:

[0035] converting an initial digital domain signal corresponding to the sub-data into an analog domain signal, and inputting the analog domain signal into each second preset modulator;

[0036] Performing multi-wavelength modulation on the analog domain signal using each of the second preset modulators to obtain a modulated signal corresponding to each of the second preset modulators;

[0037] Using the wavelength division multiplexer associated with each second preset modulator, the modulated signal corresponding to each second preset modulator is separated and copied according to the signal wavelength to obtain a demultiplexed signal;

[0038] A delay line is used to perform dislocation processing on the demultiplexed signal to obtain the first optical domain signal.

[0039] In one possible implementation, the method further includes the step of calibrating the weight of the phase change material characterization:

[0040] Initializing each phase change material in the optical neural network to a light signal all-pass state, and inputting a calibration light signal of the same intensity into each input channel of the optical neural network;

[0041] collecting the optical power output by each output channel corresponding to the cross-waveguide array in the optical neural network, and calibrating the minimum optical power value among the optical powers as a preset value;

[0042] Determining the target optical power of each phase change material according to the preset value, the minimum optical power, and the convolution kernel weight corresponding to each phase change material;

[0043] According to the target light power of each phase change material, each phase change material is heated respectively so that its optical constant value matches the convolution kernel weight.

[0044] In one possible implementation, the convolution kernel weights are trained as follows:

[0045] During the iterative training of the optical neural network, the convolution kernel weights quantized by the set bit value are used in the forward propagation process, and the convolution kernel weights are obtained by gradient calculation using floating point numbers in the backward propagation process.

[0046] In a second aspect, an embodiment of the present disclosure further provides a data processing device, including:

[0047] a traversal module, configured to traverse the data to be processed according to a convolution calculation rule, and for the currently traversed sub-data, encode an initial digital domain signal corresponding to the sub-data into a first optical domain signal;

[0048] a processing module, configured to input the first optical domain signal into an optical neural network for at least one convolution process and nonlinear processing to obtain a second optical domain signal of the sub-data on an output channel; a convolution kernel weight used in the convolution process being characterized by an optical constant value of a phase change material in each integrated storage and computing unit provided in the optical neural network;

[0049] The classification module is used to convert the second optical domain signals corresponding to the respective sub-data obtained through traversal into target digital domain signals and input them into the classification neural network to obtain the classification results of the data to be processed.

[0050] In a third aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium, comprising a computer program, which, when executed by a processor, implements the above-mentioned first aspect, or the steps in any possible implementation of the first aspect.

[0051] In a fourth aspect, an optional implementation of the present disclosure further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the first aspect or any possible implementation of the first aspect are implemented.

[0052] The data processing methods, devices, storage media, and computer devices provided by the embodiments of the present disclosure, because once the optical constant value of the phase change material is fixed, the convolution kernel weights can be stabilized in the off-power state. Therefore, by implementing convolution calculations in the integrated storage and computing unit of the optical neural network and using the optical constant value of the phase change material deployed in the integrated storage and computing unit to characterize the convolution kernel weights, it is possible to use weights for convolution calculations at near-zero static power consumption, thereby reducing the computational power consumption and inference latency of the optical neural network. When the optical neural network is used to perform convolution calculations on each sub-data obtained by traversal, no power is required to control the weights during each calculation. This extends the computing power of the phase change material to the entire network layer, greatly improving the inference speed of the data to be processed and reducing inference power consumption. Finally, a classification neural network is used to perform edge computing classification processing on the second optical domain signals corresponding to each sub-data, thereby obtaining accurate classification results. The in-memory computing architecture based on phase change materials proposed in this application (i.e., each integrated storage and computing unit in the cross-waveguide array) can achieve near-zero power multi-layer neural network inference, effectively improving inference response speed and reducing computational power consumption.

[0053] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1is a flow chart of a data processing method shown in an exemplary embodiment of the present application;

[0055] Figure 2 This is a structural diagram of a data encoding module shown in an exemplary embodiment of the present application;

[0056] Figure 3 This is a specific implementation flow chart of a data processing method shown in an exemplary embodiment of the present application;

[0057] Figure 4 is a schematic structural diagram of an optical neural network shown in an exemplary embodiment of the present application;

[0058] Figure 5 This is a hardware structure diagram of a terminal where a data processing device 600 is located, as shown in an exemplary embodiment of the present application;

[0059] Figure 6 FIG. 1 is a schematic diagram of a data processing device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0060] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0061] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0062] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0063] Research has found that edge computing, as a distributed computing framework, shifts data processing and analysis tasks from centralized data centers to edge devices on the network, significantly reducing data transmission latency. However, edge devices typically lack significant computing power, resulting in high response latency and high computational energy consumption for electrical neural network inference tasks. While methods for applying optical neural networks to edge computing have been proposed, conventional applications still rely on electricity to control the size of neural network elements (i.e., the size of convolution kernel weights). Due to the large number of neurons, optical neural networks still have high static power consumption, making them difficult to meet edge computing requirements. Therefore, reducing the computational latency and power consumption of optical neural networks when applied to edge computing has become a technical issue worthy of attention.

[0064] Based on the above research, the present disclosure provides a data processing method, device, storage medium and computer equipment. Since the optical constant value of the phase change material is fixed, the convolution kernel weight can be stabilized in the non-powered state. Therefore, by implementing the convolution calculation in the storage and computing integrated unit of the optical neural network, and using the optical constant value of the phase change material deployed in the storage and computing integrated unit to characterize the convolution kernel weight, it is possible to use the weight to perform convolution calculation under near-zero static power consumption, thereby reducing the computing power consumption and reasoning delay of the optical neural network. When the optical neural network is used to perform convolution calculations on each sub-data obtained by traversal, no electricity is required to control the weights when each calculation uses the weights, which realizes the expansion of the computing power of the phase change material to the level of the entire network layer, thereby greatly improving the reasoning speed of the data to be processed and reducing the reasoning power consumption. Finally, the classification neural network is used to perform classification processing on the second optical domain signal corresponding to each sub-data in the edge computing, and accurate classification results can be obtained. The in-memory computing architecture based on phase change materials proposed in this application can realize multi-layer neural network reasoning with near-zero power consumption, effectively improving the reasoning response speed and reducing computing power consumption.

[0065] The defects in the above solutions are the results obtained by the inventors after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below are all contributions made by the inventors to the present disclosure during the disclosure process.

[0066] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0067] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0068] It should be noted that the specific terms mentioned in the embodiments of the present disclosure include:

[0069] U-Net: is a convolutional neural network architecture for image segmentation, designed to handle biomedical image segmentation tasks;

[0070] P-type semiconductor: also known as hole-type semiconductor, is a semiconductor that mainly conducts electricity through positively charged holes;

[0071] N-type semiconductor: also known as electronic semiconductor, that is, an impurity semiconductor in which the concentration of free electrons is much greater than the concentration of holes.

[0072] To facilitate understanding of this embodiment, a data processing method disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the data processing method provided in the embodiment of the present disclosure is generally a terminal device or other processing device with certain computing capabilities, where the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a handheld device, a computer device, etc.; in some possible implementations, the data processing method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0073] The data processing method provided by the embodiment of the present disclosure is described below using the edge-side device as an example of the execution subject.

[0074] like Figure 1 FIG. 1 is a flowchart of a data processing method provided by an embodiment of the present disclosure, which may include the following steps:

[0075] S101: traversing the data to be processed according to the convolution calculation rule, and encoding the initial digital domain signal corresponding to the sub-data currently traversed into a first optical domain signal.

[0076] Here, the data processing method provided in the embodiment of the present application can be applied to edge computing tasks in edge-side devices, such as image classification tasks, text processing tasks, etc.

[0077] An optical neural network can be a pre-trained neural network based on optical computing. The convolution calculation rules specify information such as the sliding window size and sliding step size during convolution processing. For example, using a 3×3 sliding window size, the data to be processed can be traversed according to the 3×3 data block size and sliding step size to obtain the various sub-data.

[0078] The data to be processed is related to the edge computing task of the specific application of the optical neural network. For example, when the edge computing task is an image classification task, the data to be processed can be image data to be processed, video data to be processed, etc.; when the edge computing task is a text processing task, the data to be processed can be text to be processed, documents to be processed, etc.

[0079] The sub-data may be each data block obtained by each traversal. For example, when traversing an image to be processed, the data to be processed may be each image data block obtained by the traversal.

[0080] The initial digital domain signal is the digital signal of the sub-data in the digital domain, and the first optical domain signal is the optical domain signal in the optical domain. Since the data to be processed is typically a signal in the data domain, and the input to the optical neural network must be an optical domain signal, the initial digital domain signal of the sub-data needs to be converted into an optical domain signal for input into the optical neural network for data processing. For example, if the data to be processed is an image, the data corresponding to each pixel in the image is stored as a digital signal, i.e., the digital domain signal.

[0081] Exemplarily, in order to realize the convolution operation of the optical neural network on the data to be processed, it is necessary to design a corresponding data encoding module for the optical neural network to encode the data to be processed. Among them, the data encoding module includes a data encoding process and a corresponding hardware link. Specifically, after obtaining the data to be processed, the data block size required for each traversal can be determined according to the rules of the convolution calculation, and then the data to be processed is traversed according to the data block size to obtain the sub-data block obtained in each traversal, and the data corresponding to the sub-data block is used as the sub-data. During the traversal process, for the sub-data currently traversed, the initial digital domain signal corresponding to the sub-data can be encoded into a first optical domain signal through a pre-designed data encoding process and a corresponding hardware link.

[0082] Optionally, the digital domain-optical domain conversion technology in the prior art may be used to encode the initial digital domain signal of the currently traversed sub-data into the first optical domain signal.

[0083] In one embodiment, the data encoding module in the present application may include a plurality of second preset modulators and a wavelength division multiplexer (DMUX) associated with each second preset modulator. The second preset modulator is a modulator for optically modulating an analog domain signal. Exemplarily, the second preset modulator may be a Mach-Zehnder Modulator (MZM). The number of second preset modulators is related to the number of input channels and the size of the convolution kernel of the cross-waveguide array. For example, when the number of input channels of the cross-waveguide array is 9, the convolution kernel size is 3×3, and the second preset modulator has 3 wavelengths, the number of second preset modulators may be 3; when the number of input channels is 16, the convolution kernel size is 4×4, and the second preset modulator has 4 wavelengths, the number of second preset modulators may be 4. The step of "encoding the initial digital domain signal corresponding to the sub-data into the first optical domain signal" in the above S101 will be explained below in conjunction with the specific structure of the data encoding module:

[0084] S101 - 1 : Converting an initial digital domain signal corresponding to the sub-data into an analog domain signal, and inputting the analog domain signal into each second preset modulator.

[0085] Here, the analog domain signal may be an analog signal in an electrical analog domain, and the second preset modulator may be an MZM.

[0086] In specific implementations, when traversing the data to be processed according to the convolution calculation rules, the data to be processed can be sliced. Each slice is the sub-data corresponding to the sub-data block obtained in each traversal. Each row of the sub-data obtained in each traversal can then be input into an MZM in the data encoding module for electro-optical modulation.

[0087] Taking the case where the sub-data is a data block of 3×3 size and MZM includes 3 as an example, the analog domain signal of the first row of data of the 3×3 data block can be input into the first MZM for electro-optical modulation; the analog domain signal of the second row of data can be input into the second MZM for electro-optical modulation; and the analog domain signal of the third row of data can be input into the third MZM for electro-optical modulation.

[0088] S101 - 2 : Perform multi-wavelength modulation on the analog domain signal using each second preset modulator to obtain a modulated signal corresponding to each second preset modulator.

[0089] Here, the number of wavelengths in the multi-wavelength modulation can be related to the number of input channels of the cross-waveguide array and / or the convolution kernel size and / or the network design structure. For example, when the number of input channels of the cross-waveguide array is 9 and the convolution kernel size is 3×3, the number of wavelengths can be 3; when the number of input channels of the cross-waveguide array is 16 and the convolution kernel size is 4×4, the number of wavelengths can be 4. After the second preset modulator performs multi-wavelength modulation on the input analog domain signal, it can output a modulated signal including multiple wavelengths. Since each row of the analog domain signal corresponding to a sub-data will be input to a different second preset modulator for electro-optical modulation, each second preset modulator can output a modulated signal related to the sub-data.

[0090] In a specific implementation, after the analog domain signal corresponding to the sub-data is input into each second modulator, for each second preset modulator, the second preset modulator can perform multi-wavelength modulation on the signal intensity of the input analog domain signal to obtain a modulated signal including multiple wavelengths.

[0091] S101 - 3 : Using the wavelength division multiplexer associated with each second preset modulator, separate and copy the modulated signal corresponding to each second preset modulator according to the signal wavelength to obtain a demultiplexed signal.

[0092] In specific implementation, a wavelength division multiplexer is associated with each second preset modulator, and the modulated signals output by each second preset modulator can be input into the associated wavelength division multiplexer. The wavelength division multiplexer is used to separate and copy the modulated signals according to the signal wavelength and the number of wavelengths of the modulated signal to obtain multiple demultiplexed signals.

[0093] For example, a modulated signal includes three wavelengths. After separation and replication using a wavelength division multiplexer, three demultiplexed signals can be obtained. The three demultiplexed signals have the same light intensity but different wavelengths.

[0094] S101 - 4 : Use a delay line to perform shift processing on the demultiplexed signal to obtain a first optical domain signal.

[0095] In specific implementation, for each demultiplexed signal, a delay line can be used to delay the demultiplexed signal, so that the demultiplexed signals of different wavelengths are staggered in time to obtain a staggered signal. The obtained staggered signal is used as the first optical domain signal and input into the optical neural network for processing.

[0096] For example, after the modulated signal output by a second preset modulator is processed by the wavelength division multiplexer and the delay line, multiple offset signals with different wavelengths but the same intensity can be obtained.

[0097] In this way, the data encoding process of the present application utilizes the translation-invariant property of convolution, reduces the number of second preset modulators required, and reduces the power consumption of data conversion. For example, if the number of input channels of the cross-waveguide array is 9 and the convolution kernel size is 3×3, if the data encoding process of the present application is not used, 9 second preset modulators are required for electro-optical modulation. However, the present application only requires 3 second preset modulators, effectively reducing the number of second preset modulators used and reducing conversion power consumption.

[0098] like Figure 2 FIG. 1 is a structural diagram of a data encoding module provided in an embodiment of the present application, wherein Figure 2 In this paper, we take the cross waveguide array as an example with 9 input channels and 3×3 convolution kernel size. ~ Represents the initial digital domain signal corresponding to the pixel point in the first row and first column of the data to be processed to the initial digital domain signal corresponding to the pixel point in the mth row and nth column; the values ​​of m and n are related to the size of the data to be processed; Figure 2 Only the initial digital domain signals corresponding to some pixels are shown in FIG, and the initial digital domain signals corresponding to the remaining pixels are represented by ellipsis. After traversing the data to be processed, each slice can be obtained, that is, each sub-data can be obtained. Figure 2 Only three sub-data obtained after three traversals are shown in FIG. 3 , but in actual application, the number of sub-data obtained is determined according to the size of the data to be processed. Figure 2 Only three sub-data (sub-data 1 to 3) are used as examples. Each sub-data includes 9 pixels. Figure 2 In each sub-data Indicates the intensity of the analog domain signal corresponding to the pixel in the sub-data, where the value of a is less than or equal to m, and the value of b is less than or equal to n. For example, Figure 2 In sub-data 1 Represents the pixel points in the sub-data The corresponding analog domain signal strength, Represents the pixel points in the sub-data The intensity of the corresponding analog domain signal. For sub-data 1, the three lines of data can be input into three Mach-Zehnder modulators (i.e., MZM-1~MZM-3) for multi-wavelength modulation to obtain three modulated signals. Then, the modulated signals output by MZM-1~MZM-3 are input into the corresponding associated wavelength division multiplexer (DMUX) for optical signal separation and replication, and a time-shifted signal is formed through a delay line to obtain 9-way shifted signals. The 9-way shifted signals are used as the first optical domain signal corresponding to sub-data 1 and input into the optical neural network for processing. For example, for Figure 2The nine staggered signals in the figure can be the nine signals in the same column of each staggered signal (such as Figure 2 In the fifth column ) are multiplied and accumulated with the 9 weights of the convolution kernel in the optical neural network to realize the convolution processing of a convolution kernel. Figure 2 In the example, MAM-1 corresponds to three staggered signals, i.e., three signals with different wavelengths. ; MAM-2 corresponds to three staggered signals, i.e. three signals with different wavelengths ; MAM-3 corresponds to three-way misaligned signals, i.e. three signals with different wavelengths .exist Figure 2 Figure 2 shows a cross-waveguide array for convolution processing in an optical neural network, where each row represents a convolution kernel, and each square in each row represents a storage and computation unit corresponding to the convolution kernel.

[0099] In this way, using the translation invariance of convolution, Figure 2 By using 3 MZMs instead of 9 MZMs, 9-way staggered signals can be obtained, and convolution calculations can be performed with the 9 weights of the convolution kernel respectively, which effectively reduces the power consumption of data conversion.

[0100] It should be noted that this application uses the optical neural network to reason with the sub-data obtained in each traversal, and only after obtaining the second optical domain signal corresponding to the sub-data will the optical neural network be used to reason with the sub-data obtained in the next traversal. Figure 2 In the process, after traversing to obtain sub-data 1, encoding the initial digital domain signal corresponding to sub-data 1 into the first optical domain signal and using the optical neural network to infer the second optical domain signal of sub-data 1, the next traversal will be performed to obtain sub-data 2, encoding the initial digital domain signal corresponding to sub-data 2 into the first optical domain signal and using the optical neural network to infer the second optical domain signal of sub-data 2, and then the next traversal will be performed to obtain sub-data 3, and so on, until the second optical domain signal corresponding to the last sub-data is obtained.

[0101] S102: Input the first optical domain signal into the optical neural network for at least one convolution process and nonlinear processing to obtain a second optical domain signal of the sub-data on the output channel; the convolution kernel weight used in the convolution process is characterized by the optical constant value of the phase change material in each storage and computing integrated unit set in the optical neural network.

[0102] Here, the optical neural network can be deployed as a whole in the optical computing chip, and can accelerate the feature extraction process of the first optical domain signal to obtain the second optical domain signal corresponding to the sub-data. The above-mentioned output channel can be the output channel of the last network layer of the optical neural network. The convolution kernel weights can be the individual network weights corresponding to the convolution kernel, and the data of the network weights corresponding to a convolution kernel can be determined according to the network structure and / or convolution requirements of the convolution layer. For example, when the convolution kernel size is 3×3, the convolution kernel corresponds to 9 network weights. The optical constant value of the phase change material can specifically be the crystallization state of the phase change material.

[0103] To address the high static power consumption problem of existing optical neural networks, the present invention proposes an in-memory computing architecture based on phase change materials, which can achieve near-zero power consumption inference at each layer of neural network. Specifically, the in-memory computing structure can include at least one convolutional layer (i.e., a cross-waveguide array referred to below) and a nonlinear layer (i.e., a micro-ring modulation array referred to below) connected to each convolutional layer. The number of convolutional layers can be determined based on the task requirements and scale of the optical neural network, and is not specifically limited in the embodiments of this application. The first optical domain signal can be sequentially processed through each convolutional layer and nonlinear layer for convolution and nonlinear processing, respectively, to obtain a second optical domain signal corresponding to the sub-data, thereby completing feature extraction.

[0104] Since phase change materials are non-volatile, they are in a crystalline or amorphous state after applying different thermal drives. Their optical constant values ​​change with the state and are used to simulate weight values. Once the optical constant values ​​of the phase change material are determined by thermal drive, they will not change even when the power is off. Therefore, once the optical constant values ​​of the phase change material are set, the corresponding network weights can be maintained without continuous power supply during subsequent use. Among them, the network weights are the convolution kernel weights of the optical neural network. Once the optical constant values ​​of the phase change material are fixed, they can still be used to characterize the convolution kernel weights even when the power is off.

[0105] An optical neural network can be equipped with multiple integrated storage and computation units, each used for a weight multiplication operation. The number of integrated storage and computation units is related to the size and number of convolution kernels. For example, if the convolution kernel size is 3×3 and the number of convolution kernels is 9, there can be 81 integrated storage and computation units. The integrated storage and computation units include phase change materials, and the optical constant value of the phase change material is used to represent a weight of a convolution kernel.

[0106] In a specific implementation, each first optical domain signal corresponding to the sub-data and matching the number of input channels of the cross-waveguide array can be input into an optical neural network, and each first optical domain signal can be convolved using each integrated storage and computing unit in the optical neural network. The convolution result is then nonlinearly processed using a nonlinear layer to obtain the second optical domain signal of the sub-data on the output channel. The weighted multiplication operation in the convolution operation can be simulated by energy attenuation of the first optical domain signal using a phase change material. For example, the convolution kernel weight represented by the optical constant value of the phase change material in the integrated storage and computing unit can be used to perform multiplication processing on the first optical domain signal.

[0107] In this way, the optical neural network is used to perform convolution and nonlinear processing on the first light domain signal corresponding to the sub-data obtained in each traversal, thereby realizing feature extraction of the first light domain signal and obtaining the second light domain signal corresponding to the sub-data obtained in each traversal.

[0108] This application considers phase-change materials as non-volatile storage materials, allowing neural network weights to be stored in optical neural networks without power. Phase-change materials can switch between crystalline and amorphous states, with different states causing significant changes in the values ​​of optical constants, which can represent the weights of convolution kernels in the neural network. Introducing phase-change materials into optical neural networks can effectively reduce the high static power consumption of optical computing chips, potentially meeting the energy requirements of edge computing. Prior research on using phase-change materials for optical computing is in its early stages, and the technical feasibility of using phase-change materials for optical computing has been demonstrated at the level of individual operators, such as matrix multiplication and convolution. This application extends the computing power of phase-change materials at the operator level to the entire network level (for example, an N-layer optical neural network can be implemented using phase-change materials deployed on an optical computing chip). The application designs an optical deep neural network computing architecture to achieve near-zero power consumption for optical neural network inference. For simplicity, prior art can only perform computations on single operators, such as matrix multiplication; at the network level, this is equivalent to constructing an optical neural network using phase-change materials, and using phase-change materials to perform convolution computations within the optical neural network. In this application, phase change materials are used to implement convolution calculations of multiple hidden layers, thereby realizing the application of phase change materials at the network level.

[0109] S103: Convert the second optical domain signals corresponding to the sub-data obtained through the traversal into target digital domain signals and input them into the classification neural network to obtain the classification results of the data to be processed.

[0110] Here, the classification neural network can be a trained electrical neural network used to perform classification tasks. The input to the classification neural network must be a digital signal. Therefore, after obtaining the second optical domain signal corresponding to each sub-data, the second optical domain signal must be converted into a digital signal for processing by the classification neural network. The classification result indicates the classification processing result of the data to be processed in the edge computing classification task.

[0111] During specific implementation, the second optical domain signal corresponding to each sub-data can be converted into an electrical signal through a photoelectric detector (PD), and then the electrical signal is encoded into a target digital domain signal. The target digital domain signal corresponding to each sub-data is then input into a classification neural network to obtain a classification result of the data to be processed. Among them, the process of encoding the electrical signal into the target digital domain signal can be to first encode the electrical signal into an analog domain signal, and then convert the analog domain signal into a target digital domain signal. In this way, the second optical domain signal output from the output end of the optical neural network is subjected to photoelectric conversion and analog-to-digital conversion in sequence to obtain a target digital domain signal that can be processed by the classification neural network. Optionally, the classification neural network can be a classification network in the prior art.

[0112] The optoelectronic hybrid neural network proposed in this application modulates the data to be processed into the optical domain through electro-optical conversion. The optical domain signal is then passed through the optical neural network for high-speed, low-power feature extraction. The output signal of the optical neural network is then converted back into the electrical digital domain and passed through the back-end classification neural network to complete the data classification task.

[0113] like Figure 3 The flowchart of a data processing method provided by the present application may include the following steps:

[0114] Data to be processed ( Figure 3 (Using the image to be processed as an example) the data is modulated into the optical domain through electro-optical conversion and input into the optical neural network at the front end for feature extraction to obtain the optical domain signal. The optical domain signal is input into the classification neural network at the back end for classification processing to obtain the classification result. Figure 3 , the classification result is the 9th of the 9 results. In this way, the optoelectronic hybrid neural network computing architecture proposed in this application can reduce the network's inference latency and computing power consumption, thereby meeting the edge-side computing performance requirements. Specifically, the optoelectronic hybrid network is divided into two parts, the front-end and the back-end. The front-end network extracts the features of the data to be processed, which is implemented by the optical neural network, and the back-end network performs feature classification, which is implemented in the electrical digital domain by a classification neural network.

[0115] This application targets the network classification task in edge computing by dividing the network into two parts: a front-end optical neural network and a back-end classification neural network. The front-end network completes the feature extraction of the data, and the back-end network performs classification based on the features extracted by the front-end network. The storage and computing integrated optical computing chip designed in this application can be used for data processing in the front-end network to accelerate the feature extraction process and reduce the computing power consumption of this part. The overall data processing flow is as follows: (1) Extract the data in the image in sequence in the electrical domain; (2) Convert the electrical domain digital signal into an optical domain analog signal through digital-to-analog conversion and electro-optical conversion; (3) The optical domain analog signal completes the convolution operation and nonlinear processing of the image through the optical neural network to obtain a modulated optical signal; (4) Convert the modulated optical signal output by the optical neural network into an electrical domain digital signal through photoelectric conversion and analog-to-digital conversion; (5) The electrical domain digital signal completes the task classification through the back-end classification neural network.

[0116] In one embodiment, an optical neural network may include at least one hidden layer, wherein each hidden layer includes a cross-waveguide array for convolution processing and a micro-ring modulation array for nonlinear processing; wherein the micro-ring modulation array is connected after the cross-waveguide array, and the cross-waveguide array includes multiple storage and computing units. A cross-waveguide array is equivalent to a convolution layer, which can implement convolution processing of data and complete feature extraction of the data to be processed; a micro-ring modulation array is equivalent to a nonlinear layer, which can implement nonlinear processing of data; that is, the convolution layer is implemented through the network architecture of the cross-waveguide array, and the nonlinear layer is implemented through the micro-ring modulation array. The structure of the micro-ring modulation array is related to the structure of the cross-waveguide array. For example, for each row of the cross-waveguide array, there is a corresponding row in the micro-ring modulation array. This application supports the construction of multiple hidden layers in the optical neural network, thereby increasing the depth of the optical neural network.

[0117] The number of hidden layers is related to the task requirements and network scale of the optical neural network. Within each hidden layer, two adjacent hidden layers are connected, and the output of the previous hidden layer serves as the input to the next hidden layer. The output of the last hidden layer is the second optical domain signal. The convolution kernel sizes corresponding to the cross-waveguide arrays in different hidden layers can vary. The cross-waveguide array in each hidden layer has a set number of input channels and a set number of output channels. The number of input channels and output channels of the cross-waveguide arrays in different hidden layers can differ, or the number of input channels of the previous cross-waveguide array can be the same as the number of output channels of the next cross-waveguide array, while the number of output channels of the next cross-waveguide array can differ from that of the previous cross-waveguide array. In this way, different hidden layers can be used to expand or compress the number of feature map channels of the data to be processed.

[0118] A hidden layer can also be called a network layer, which is used to perform one convolution process and one nonlinear process on the processed data. Therefore, the number of convolution processes and nonlinear processes performed by the optical neural network on the processed data is related to the number of hidden layers in the optical neural network.

[0119] For example, an optical neural network can implement the data processing flow in existing convolutional neural networks such as U-Net, perform convolution and nonlinear processing on the data to be processed, and obtain processing results. Specifically, the optical neural network can include two hidden layers. The cross-waveguide array in hidden layer 1 can be used to expand the number of channels of the data to be processed. After expansion, convolution with different convolution kernels in the cross-waveguide array can obtain a variety of convolution features, thereby improving the diversity of convolution features; the cross-waveguide array in hidden layer 2 can be used to compress the number of channels of the data to be processed, thereby enabling the optical neural network to eliminate redundant feature information and reduce the pressure of photoelectric conversion and analog-to-digital conversion at the output end of the optical neural network.

[0120] Regarding the above S102, it can be implemented according to the following steps:

[0121] S102-1: Input the first light domain signal into the first hidden layer in the optical neural network, and use the storage and computing integrated units in the first hidden layer to perform convolution processing on the first light domain signal to obtain a convolution result.

[0122] In specific implementation, after obtaining the first optical domain signal of the sub-data, the first optical domain signal can be input into the cross-waveguide array in the first hidden layer of the optical neural network, and the first optical domain signal can be convolved using the various storage and computing units included in the cross-waveguide array in the first hidden layer (i.e., completing the point product operation of the first optical domain signal and the convolution kernel weight), thereby realizing feature extraction of the first optical domain signal and obtaining the convolution result.

[0123] S102-2: Input the convolution result into the micro-ring modulation array of the first hidden layer for nonlinear processing to obtain a nonlinear result.

[0124] In a specific implementation, the convolution result output by the cross-waveguide array in the first hidden layer can be input into the micro-ring modulation array in the first hidden layer for nonlinear processing to obtain a nonlinear result, wherein the nonlinear result is a signal in the optical domain.

[0125] For example, the nonlinear processing of the present application can be implemented in an optical-electrical-optical manner. Specifically, the convolution result can be converted into a current signal through a photodetector, and the current signal is used to drive the carrier concentration of the microring modulator in the microring modulation array, thereby realizing the modulation of the resonance curve. The nature of the resonance curve determines that the response relationship between the modulated electrical signal and the output optical signal is nonlinear, thereby obtaining a nonlinear result.

[0126] S102-3: If there is a next hidden layer, input the nonlinear result to the next hidden layer to obtain a new nonlinear result output by the next hidden layer.

[0127] When an optical neural network includes multiple hidden layers, each hidden layer processes the data to be processed sequentially. Therefore, after obtaining a nonlinear result using the micro-ring modulation array of the previous hidden layer, if the next hidden layer exists, the nonlinear result output by the previous hidden layer can be input into the next hidden layer for convolution and nonlinear processing to obtain a new nonlinear result. If the next hidden layer does not exist, the nonlinear result output by the current hidden layer can be used as the second optical domain signal of the sub-data on the output channel.

[0128] Exemplarily, after the first hidden layer is processed, if a next hidden layer exists, the nonlinear result obtained from the first hidden layer processing is input into the next hidden layer for processing, thereby obtaining a nonlinear result output by the next hidden layer. The processing of the next hidden layer also involves first performing convolution processing on the nonlinear result output by the previous hidden layer using the cross-waveguide array in the hidden layer, and then performing nonlinear processing on the convolution result using the micro-ring modulation array in the hidden layer, thereby obtaining a new nonlinear result output by the hidden layer. The cross-waveguide arrays in different hidden layers can have different array sizes, and the micro-ring modulation arrays can also have different array sizes.

[0129] S102-4: Use the nonlinear result output by the last hidden layer as the second optical domain signal of the sub-data on the output channel.

[0130] In a specific implementation, after obtaining the nonlinear output of a hidden layer, if the next hidden layer exists, the nonlinear output can be input into the next hidden layer for processing. If the next hidden layer does not exist, indicating that the current hidden layer is the last hidden layer, the nonlinear output of the last hidden layer can be used as the second optical domain signal of the sub-data on the output channel. The number of output channels of the cross-waveguide array in the last hidden layer is the same as the number of output channels at the output end of the optical neural network.

[0131] like Figure 4The figure is a schematic diagram of the structure of an optical neural network provided by the embodiment of the present application. Figure 4 The content of this article explains the processing of optical neural networks:

[0132] by Figure 4 For example, the optical neural network in

[15] includes two hidden layers. The first hidden layer may include convolution layer 1 (i.e., cross waveguide array 1) and nonlinear layer 1 (i.e., microring modulation array 1). The second hidden layer may include convolution layer 2 (i.e., cross waveguide array 2) and nonlinear layer 2 (i.e., microring modulation array 2). The second hidden layer is located after the first hidden layer. Nonlinear layer 1 and nonlinear layer 2 are both used for nonlinear processing, i.e., to perform In convolutional layer 1, the dimension of the cross-waveguide array is 9×9, which can simultaneously implement convolution operations with 9 different convolution kernels, where the convolution kernel size is 3×3. The number of input channels of cross-waveguide array 1 is 9, and the number of output channels is 9. Figure 4 The k1~k9 in the convolution layer 1 in the cross waveguide array are the 9 convolution kernels of the convolution layer 1. Each row of the cross waveguide array represents the convolution kernel weight corresponding to a convolution kernel. Since the number of rows of the cross waveguide array 1 is 9, the convolution operation of 9 different convolution kernels can be realized in parallel. Figure 4 After the image to be processed is obtained, the image to be processed can be traversed in a 3×3 data block size, and each traversal obtains a 3×3 sub-data block. The initial digital domain signal corresponding to the sub-data block is encoded into a first optical domain signal. In this way, a first optical domain signal composed of optical domain signals corresponding to the 9 pixels in the sub-data block can be obtained. Figure 4 λ1 to λ9 below convolution layer 1 in the image represent the wavelengths of the optical signals corresponding to the nine pixels. The first optical signal passes through cross-waveguide array 1, performing a dot product operation between the small block of data and the convolution kernel weights to obtain the convolution result. This convolution result is then input into nonlinearity 1 for nonlinear processing, resulting in the nonlinear result after the first hidden layer processing. Figure 4 The dimension of convolution layer 2 in the image is 9×1. There is one convolution kernel in convolution layer 2 with a size of 1×1. The number of input channels of cross waveguide array 2 is 9, and the number of output channels is 1. The nonlinear result after processing by the first hidden layer is input to cross waveguide array 2 for convolution processing to obtain a convolution result. This convolution result is then input to nonlinearity 2 for nonlinear processing to obtain a nonlinear result. This nonlinear result is used as the second optical domain signal of the sub-data on the output channel.

[0133] In one embodiment, each storage and computing integrated unit may include at least three parts: input and output waveguides, directional couplers, and phase change materials. The input and output waveguides may be cross-waveguides constructed using silicon nitride (SiN), which may include input waveguides and output waveguides arranged in a cross-shaped arrangement. Figure 4 In the storage-computing integrated unit, the vertical waveguide wire is used to indicate the input waveguide, and the horizontal waveguide wire is used to characterize the output waveguide. The directional coupler can be constructed using silicon Si. The state of the phase change material is controlled by an external voltage, which can heat the heater around the phase change material. The heat is then transferred to the phase change material to control the crystallization or amorphization state of the phase change material, thereby modifying the optical constant value of the phase change material. When constructing an optical neural network, the optical constant value of the phase change material in each storage-computing integrated unit can be set by applying a voltage according to the convolution kernel weight of each convolution kernel, thereby realizing the characterization of the convolution kernel weight. When applying the constructed optical neural network, the input light signal can be energy attenuated by the phase change material to simulate the weight multiplication operation in the neural network.

[0134] The multiple storage and computing integrated units included in the cross waveguide array can be arranged according to a set structure, and each storage and computing integrated unit in a row or a column of the arrangement corresponds to a convolution kernel, and each convolution kernel weight of the convolution kernel is characterized by the optical constant value of the phase change material of each storage and computing integrated unit. Here, the set structure can be a cross-cross structure, and each row arranged under the cross-cross structure can correspond to a convolution kernel, that is, each storage and computing integrated unit in each row corresponds to a convolution kernel, and each convolution kernel weight of the convolution kernel is characterized by the optical constant value of the phase change material of each storage and computing integrated unit, or each storage and computing integrated unit in each column corresponds to a convolution kernel, and each convolution kernel weight of the convolution kernel is characterized by the optical constant value of the phase change material of each storage and computing integrated unit. As for whether a row corresponds to a convolution kernel or a column corresponds to a convolution kernel under the set structure arrangement, it can be designed according to the specific network structure, and the embodiments of the present application do not make specific limitations. For example, in Figure 4 In the figure, the dimension of the cross-waveguide array 1 is 9×9, with a total of 9 convolution kernels, and the size of the convolution kernel is 3×3, that is, Figure 4 The cross-waveguide array 1 in the convolution layer is a 9-row × 9-column array. Each row in the array corresponds to a convolution kernel. The optical constant value of the phase change material in the 9 storage-computation integrated units in each row is used to characterize the 9 convolution kernel weights of the convolution kernel corresponding to the row. The cross-waveguide array structure in the convolution layer 2 is the same as that in the convolution layer 1, with a dimension of 9 × 1 and a convolution kernel size of 1 × 1. The number of input channels of the cross-waveguide array 2 is 9 and the number of output channels is 1, that is, Figure 4 The cross-waveguide array 2 is an array of 9 rows and 1 column, each column in the array corresponds to a convolution kernel, and the optical constant value of the phase change material in the 9 storage-computation integrated units in each column is used to characterize the 9 convolution kernel weights of the convolution kernel corresponding to the column.

[0135] Regarding the above S102-1, it can be implemented as follows:

[0136] S102-1-1: For any convolution kernel of the first hidden layer, the first optical domain signal is input to each storage and computing integrated unit corresponding to the convolution kernel through each input waveguide corresponding to the convolution kernel.

[0137] In a specific implementation, the first optical domain signal corresponding to the sub-data can be input into each row of the cross-waveguide array in the first hidden layer. For each row, the first optical domain signal can be input into each storage and computing integrated unit included in the row through the input waveguides in each storage and computing integrated unit included in the row.

[0138] For example, in Figure 4 In the example, the first optical domain signal of the 3×3 sub-data block of the image to be processed (i.e., the optical domain signals corresponding to λ1 to λ9 below the convolution layer 1) can be input into the 9 integrated storage and computing units in each row of the cross waveguide array 1 through the input waveguides in the integrated storage and computing units in each row of the cross waveguide array 1. For example, using Figure 4 The input waveguides of the nine storage and computing integrated units in the first row of the cross waveguide array 1 are used to input the optical domain signals corresponding to the first optical domain signals at λ1 to λ9 into the nine storage and computing integrated units in the first row respectively; Figure 4 The input waveguides of the 9 storage-computing integrated units in the second row of the cross waveguide array 1 input the optical domain signals corresponding to λ1~λ9 of the first optical domain signal into the 9 storage-computing integrated units in the second row respectively; in this way, the optical domain signals corresponding to λ1~λ9 of the first optical domain signal can be input into the storage-computing integrated units corresponding to the 9 rows of the cross waveguide array 1 respectively.

[0139] S102-1-2: For any integrated storage and computing unit, the directional coupler in the integrated storage and computing unit is used to couple the light energy of the first optical domain signal at a set ratio, and the convolution kernel weight represented by the optical constant value of the phase change material is used for multiplication processing to obtain the multiplication result of the integrated storage and computing unit and output it through the output waveguide.

[0140] Here, the set ratio can be the ratio of the total energy of the input optical domain signal, and its size can be related to the number of convolution kernels. For example, the set ratio can be 1 / the number of convolution kernels. The reason why the optical energy of the first halo signal at the set ratio needs to be coupled is to enable the optical domain signals of each input channel to be evenly distributed to multiple storage and computing units through the directional coupler array to achieve signal replication. Among them, the directional coupler array is an array composed of directional couplers included in each storage and computing unit in the cross waveguide array. For example, in Figure 4In the example, the optical domain signal corresponding to λ1 in the cross-waveguide array 1 can be evenly distributed to the storage and computing integrated units corresponding to the first column through the directional coupler array. The optical domain signal corresponding to λ2 in the cross-waveguide array 1 can be evenly distributed to the storage and computing integrated units corresponding to the second column through the directional coupler array.

[0141] In a specific implementation, for any storage-computing integrated unit corresponding to any convolution kernel, the directional coupler in the storage-computing integrated unit can be used to couple the light energy of the first light domain signal at a set ratio, and the convolution kernel weight represented by the optical constant value of the phase change material in the storage-computing integrated unit can be used to attenuate the light energy coupled by the directional coupler, thereby achieving a weighted multiplication operation on the first light domain signal, obtaining the multiplication result and outputting it through the output waveguide of the storage-computing integrated unit. For example, Figure 4 In the embodiment, for the storage-computing integrated unit in the first row and first column of the cross-waveguide array 1, 1 / 9 of the optical energy of the optical domain signal corresponding to λ1 can be coupled through the directional coupler in the storage-computing integrated unit, and the convolution kernel weight represented by the optical constant value of the phase change material in the storage-computing integrated unit is used to perform weighted multiplication on the optical energy coupled by the directional coupler, and the multiplication result is obtained and output using the output waveguide in the storage-computing integrated unit to obtain the multiplication result under the convolution kernel corresponding to λ1 in the first row.

[0142] S102-1-3: Determine the initial convolution result corresponding to the first optical domain signal under the convolution kernel based on the multiplication result of the output waveguide outputs of each storage and computing integrated unit corresponding to the convolution kernel.

[0143] During specific implementation, for any convolution kernel corresponding to any cross-waveguide array, the output channel corresponding to the convolution kernel can be used to accumulate the multiplication results of the output waveguide outputs of each storage-computation integrated unit corresponding to the convolution kernel to obtain the initial convolution result of the first optical domain signal under the convolution kernel.

[0144] For example, Figure 4 In the figure, for the convolution kernel corresponding to the first row in the cross-waveguide array 1, the convolution kernel can correspond to an output channel, accumulate the multiplication results of the output waveguide outputs of the storage-computation integrated unit in the first row, and obtain the initial convolution result of the first optical domain signal under the convolution kernel corresponding to the first row.

[0145] Thus, the input end of the cross-waveguide array includes multiple input channels to load multi-dimensional vectors, e.g. Figure 4The nine input channels of the cross-waveguide array 1 are used to load the first optical domain signal corresponding to the nine pixels in the sub-data block. The first optical domain signal is evenly distributed to each storage and computing unit through a directional coupler to complete the signal replication. The replicated signal is subjected to energy attenuation according to the convolution kernel weight represented by the phase change material, completing the multiplication operation of the signal and the convolution kernel weight. Finally, the output end of the cross-waveguide array includes multiple output channels, each of which collects the input signals modulated by the storage and computing unit, thereby completing the accumulation operation, that is, the output channel performs incoherent superposition of the input channel signals modulated by the storage and computing unit to achieve the accumulation operation. Combining the above operations, the cross-waveguide array realizes the multiplication operation of the matrix (for example, the weight matrix corresponding to the 9×9 phase change material in the cross-waveguide array 1) and the vector. This application utilizes the characteristics of local information processing and streaming computing of convolution calculation. Each time, only a small piece of information in the data to be processed needs to be sent to the optical neural network for processing, without having to send all the data to be processed into the network at one time, thereby reducing the requirements for the channel scale of the optical neural network.

[0146] S102-1-4: Determine a convolution result of the first light domain signal according to the initial convolution results corresponding to the first light domain signal under each convolution kernel.

[0147] During specific implementation, for the first hidden layer, the initial convolution results under each convolution kernel of the cross waveguide array 1 in the first hidden layer can be used as the convolution results of the first optical domain signal in the first hidden layer.

[0148] For example, in Figure 4 In the example, for the crossed waveguide array 1, the initial convolution results of the first optical domain signal under the convolution kernel corresponding to each row of the crossed waveguide array can be obtained. That is, the initial convolution results of the first optical domain signal under the convolution kernel corresponding to nine rows are obtained. The initial convolution results under the convolution kernel corresponding to these nine rows can then be used as the convolution results of the first optical domain signal under the crossed waveguide array 1. The convolution results of the first optical domain signal under the crossed waveguide array 1 include nine convolved optical signals.

[0149] For the cross-waveguide array in any hidden layer in the optical neural network, its structure is similar to the structure of the cross-waveguide array in the first hidden layer introduced above. The difference lies in the number and size of convolution kernels and the number of input and output channels. Therefore, the process of convolution processing of the optical domain signal by the cross-waveguide array in any hidden layer in the optical neural network can be similar to the process of convolution processing of the optical domain signal by the cross-waveguide array in the first hidden layer introduced above, and this application will not go into details here.

[0150] Still Figure 4For example, assuming the shape of the input image to be processed is C×D, it will be converted into three-dimensional cubic data after processing by the cross waveguide array 1, and the data shape is C×D×9, where 9 is the number of convolution kernels in the cross waveguide array 1. The array structure of the cross waveguide array 2 is the same as that of the cross waveguide array 1, with a dimension of 9×1, a convolution kernel size of 1×1, 9 input channels, and 1 output channel. The processing flow of the cross waveguide array 2 can be as follows: the 9 input channels of the cross waveguide array 2 respectively receive the optical signals output by the 9 output channels of the hidden layer 1 (in Figure 4 Specifically, it can be the nonlinear results of the 9 channels output by the nonlinear 1); the 9 storage and computing units corresponding to a convolution kernel in the cross waveguide array 2 are used for convolution processing to obtain the convolution result under one output channel. After the cross waveguide array 2, data compression can be performed in the channel dimension, that is, the number of feature map channels is compressed to compress the data of the shape of C×D×9 output by the cross waveguide array 1 into data of the shape of C×D. That is, after Figure 4 The convolution layer 1 in the image can expand one input image into nine images, and the convolution layer 2 can compress the nine images output by the convolution layer 1 into one image.

[0151] This application uses a crossed waveguide array for convolution operations, which can be implemented by multiplying a matrix by a vector. During convolution processing, the convolution kernel size is typically small, so this application does not require the design of a large-scale crossed waveguide array to store weight parameters (i.e., it does not require the establishment of a large-scale integrated storage and computation unit to use the optical constant values ​​of the phase change material in the integrated storage and computation unit to represent the network weights). Therefore, it can adapt to the computational scale of current optical neural networks.

[0152] In one embodiment, the micro-ring modulation array includes a plurality of micro-ring modulators, each of which is provided with a doped region, wherein the doped region includes P-type semiconductor carriers and N-type semiconductor carriers. Here, the number of micro-ring modulators included in the micro-ring modulation array in each hidden layer is related to the number of convolution kernels and / or output channels of the cross-waveguide array in the hidden layer. For example, in Figure 4 In the figure, each row of convolution kernels in the cross-waveguide array 1 is connected to a micro-ring modulator (i.e., if the cross-waveguide array 1 outputs nine data, there are nine micro-ring modulators). These micro-ring modulators form a nonlinear micro-ring modulation array 1. Each micro-ring modulator has a PN-doped region. The P region in the PN-doped region includes P-type semiconductor carriers, and the N region includes N-type semiconductor carriers.

[0153] Regarding the above S102-2, it can be implemented as follows:

[0154] S102-2-1: Convert the convolution result into a current signal and input it into each micro-ring modulator in the micro-ring modulation array of the first hidden layer.

[0155] Here, the micro-ring modulation array in the present application can realize nonlinear calculation of signals, and the nonlinear calculation adopts an optical-electrical-optical implementation method.

[0156] In a specific implementation, the convolution result (the convolution result is an optical signal) output by the cross-waveguide array of the first hidden layer can be converted into a current signal through a photodetector; then the current signal is input to each micro-ring modulator in the micro-ring modulation array of the first hidden layer. Figure 4 In the example, the nine convolved optical signals output by the cross-waveguide array 1 can be converted into electrical signals to obtain nine current signals. For example, the convolved optical signal output by each row of the cross-waveguide array in the cross-waveguide array 1 can be converted into an electrical signal by a photodetector and input into the microring modulator corresponding to that row in the nonlinear system 1.

[0157] S102-2-2: Use the current signal to drive the change of the carrier concentration in the doping region, change the refractive index of the micro-ring modulator, and perform nonlinear modulation based on the changed refractive index to obtain an initial modulation result.

[0158] In a specific implementation, for each microring modulator, the voltage corresponding to the current signal input to the microring modulator can be used to drive the carrier concentration in the PN doped region of the microring modulator to change, thereby changing the refractive index of the microring modulator's material, and in turn causing the microring resonance peak of the microring modulator to shift. Because the curvilinear nature of the microring resonance peak determines the nonlinear response relationship between the modulated electrical signal and the output optical signal, the shift in the microring resonance peak will achieve nonlinear modulation of the optical signal, obtaining an initial modulation result. The modulated optical signal will serve as the input to the cross-waveguide array in the next hidden layer for convolution calculation in the next layer, thereby realizing multi-layer optical neural network reasoning.

[0159] For example, in Figure 4 In the example, the nine convolved optical signals output by the cross-waveguide array 1 have different light intensities when detected by the photodetector. Consequently, the current magnitudes of the nine current signals converted by the photodetector are also different. After being converted into nine current signals, each electrical signal can be input into a corresponding microring modulator in the nonlinear circuit 1. The voltage signal corresponding to the electrical signal modulates the refractive index of the microring modulator's material, thereby modulating the electrical signal and obtaining the initial modulation result. Since the current magnitudes of the current signals input to different microring modulators in the nonlinear circuit 1 vary, the refractive index of the material changed by each microring modulator also varies.

[0160] S102-2-3: Obtain a nonlinear result based on the initial modulation results corresponding to each micro-ring modulator.

[0161] During specific implementation, the initial modulation results corresponding to the various micro-ring modulators can be taken together as the nonlinear results output by the micro-ring modulation array.

[0162] For example, Figure 4 In the embodiment, the initial modulation results corresponding to each micro-ring modulator in the nonlinear 1 can be used as the nonlinear results output by the micro-ring modulation array corresponding to the nonlinear 1.

[0163] The optical computing chip of the present application is designed with an optical nonlinear processing unit (i.e., a micro-ring modulation array) during the inference process of each layer of the optical neural network. This nonlinear processing unit eliminates the need for any digital-to-analog conversion process for the optical signal during the convolution inference process of the optical neural network, thereby reducing computing power consumption and latency.

[0164] In one embodiment, in addition to setting up a micro-ring modulator, a first preset modulator can also be set up in the nonlinear processing part. Specifically, the micro-ring modulation array can also include a first preset modulator cascaded with each micro-ring modulator. Here, the first preset modulator can be a Mach-Zehnder modulator MZM. The cascaded use of the MZ modulator and the micro-ring modulator can increase the complexity of the nonlinear response, thereby enabling the optical neural network to have better performance. A micro-ring modulator and the first preset modulator of the cascaded micro-ring modulator can be defined as one unit in the micro-ring modulation array. For example, Figure 4 In the embodiment, nonlinearity 1 is further provided after crossed waveguide array 1, and nonlinearity 2 is further provided after crossed waveguide array 2. Nonlinearity 1 includes nine units associated with the nine output channels of crossed waveguide array 1, each unit including a microring modulator and a first preset modulator. Nonlinearity 2 includes one unit associated with one output channel of crossed waveguide array 2, and this unit includes a microring modulator and a first preset modulator.

[0165] Regarding the step of "performing nonlinear modulation based on the changed refractive index to obtain an initial modulation result" in the above S102-2-2, it can also be implemented according to the following steps 1 to 3:

[0166] Step 1: Input the current signal into the first preset modulator of each micro-ring modulator cascade, and adjust the phase of the first preset modulator.

[0167] In a specific implementation, the convolution result (which is an optical domain signal) output by the cross-waveguide array of the first hidden layer can be converted into a current signal through a photodetector. The current signal is then input into each microring modulator in the microring modulation array of the first hidden layer, as well as into the first preset modulator in cascade connection with each microring modulator. While the current signal changes the refractive index of the microring modulator, it also acts on the first preset modulator, thereby changing the phase of the first preset modulator's phase shifter.

[0168] Step 2: Using an independent laser source provided for each first preset modulator, input a preset optical domain signal to each first preset modulator respectively.

[0169] Here, the present application can equip each first preset modulator in the micro-ring modulation array with an independent laser, that is, each unit in the micro-ring position array is provided with an independent laser. By configuring an independent laser for each first preset modulator to realize the input of additional optical domain signals, the intensity of the optical domain signals output by each independent laser is the same, but the wavelength is different. For example, Figure 4 In the figure, each MZM in nonlinearity 1 is provided with an independent laser. Each independent laser inputs an additional optical domain signal to the associated MZM. The wavelength of the additional optical domain signal input by each independent laser can be λ1 to λ9 in nonlinearity 1. The additional optical domain signal here is a preset optical domain signal with a preset intensity.

[0170] In a specific implementation, while inputting the current signals corresponding to the cross-waveguide array into the micro-ring modulation array, the independent laser source provided for each first preset modulator can be used to input the preset optical domain signal into each first preset modulator respectively.

[0171] For example, Figure 4 In the example, the input of nonlinearity 1 includes not only the current signals corresponding to the cross-waveguide array 1 but also the preset optical domain signals input by the independent laser sources configured for each first preset modulator. For example, λ1 to λ9 in nonlinearity 1 correspond to different wavelengths of the preset optical domain signals.

[0172] Step 3: Using the adjusted phase and the changed refractive index, perform nonlinear modulation on the preset optical domain signal to obtain an initial modulation result.

[0173] In a specific implementation, after the current signal changes the refractive index of the micro-ring modulator and the phase of the first preset modulator, the adjusted phase and the changed refractive index can be used to perform nonlinear modulation on the preset optical domain signal input to the first preset modulator to obtain an initial modulation result. Figure 4 In the example, for each unit in Nonlinearity 1, after using a current signal to change the refractive index of the microring modulator and the phase of the MZM in that unit, the preset optical domain signal input to the MZM in that unit is nonlinearly modulated based on the adjusted phase and changed refractive index, resulting in an initial modulation result. Thus, λ1 through λ9 in Nonlinearity 1 correspond to different wavelengths of the preset optical domain signals. Although the nine preset optical domain signals have the same intensity, after being modulated by the microring modulator and the MZM, nine different initial modulation results are obtained.

[0174] For example, Figure 4The 9 optical signals after convolution in the cross-waveguide array 1 will be converted into 9 current signals. The 9 current signals will act on the corresponding units respectively, causing the refractive index of the material of the micro-ring modulator in the unit to change and the phase of the phase shifter of the MZM modulator to change (because different current signals correspond to different voltages, different voltages will lead to different refractive indices, and different refractive indices will lead to different changes in the phase of the phase shifter, so the phase change degree of the phase shifter of each MZM modulator is different). The change in refractive index and phase will affect the intensity of the preset optical domain signal input into the unit by the independent laser, thereby obtaining the initial modulation result after the final modulation. For example, Figure 4 The preset optical domain signal corresponding to λ1 in nonlinearity 1 will undergo a change in light intensity after entering the MZM. This change is caused by the phase change of the MZM phase shifter, and the phase change is caused by the current magnitude of the current signal input to the MZM. After the MZM modulates the light intensity, the resonance curve change of the microring modulator modulates the preset optical domain signal after the intensity change again, thereby obtaining the initial modulation result of the preset optical domain signal corresponding to λ1.

[0175] It can be understood that the structure of each nonlinear layer in the optical neural network (that is, each micro-ring modulation array) is similar to the structure of the nonlinear layer in the first hidden layer introduced above. The difference is that the number of units in the micro-ring modulation array is different. Therefore, the nonlinear modulation process of each nonlinear layer in the optical neural network is similar to the nonlinear modulation process of the nonlinear layer in the first hidden layer introduced above, and will not be repeated here.

[0176] In this way, in the nonlinear processing part, each unit is equipped with an independent laser for the input of an independent laser light source. On the one hand, it solves the problem of signal coherence between the output channels of the convolution layer in front of the nonlinear processing part, which causes the problem of being unable to cascade with the next-level cross-waveguide array. On the other hand, it makes the depth of the optical neural network not limited by the loss of the overall link, making it possible to build a deeper neural network structure. On the contrary, if an independent laser is not set up, and convolution layer 1 and convolution layer 2 are directly cascaded together, since there is optical coherence between the 9 optical signals output by convolution layer 1, and the 9 optical signals input by convolution layer 2 are required to be incoherent, it will cause convolution layer 2 to be unable to process or the processing effect to be unsatisfactory. By setting up an independent laser, the coherent optical signal output by convolution layer 1 can be adjusted to an incoherent optical signal, thereby meeting the input requirements of convolution layer 2.

[0177] This application designs a network structure based on a cross-waveguide array, eliminating the need for static voltage to maintain network weights, enabling convolution processing of optical neural networks with near-zero power consumption. Furthermore, by introducing a micro-ring modulation array into the optical neural network, nonlinear signal computation is achieved, enabling multi-layer deep neural network reasoning.

[0178] In one embodiment, the network weights of the optical neural network can be obtained through training. After the model training is completed, the network weights of the front-end optical neural network need to be deployed on the phase change material in each storage and computing unit in the optical computing chip. In order to improve the accuracy of weight deployment, the present invention proposes corresponding optimization strategies at the hardware level and the software level respectively to achieve calibration of the network weights corresponding to the optical constant values ​​of the phase change material, thereby alleviating the problem of decreased network reasoning accuracy as much as possible. Specifically, the hardware-level calibration can be achieved through the following steps A to D:

[0179] Step A: Initialize each phase change material in the optical neural network to a full-pass state for optical signals, and input a calibration optical signal of the same intensity into each input channel of the optical neural network.

[0180] Here, the full optical signal pass state is a state in which no optical signal is absorbed. The phase change material absorbs no light in the amorphous state and absorbs all light in the crystalline state. Therefore, it can be seen that the phase change material is in the full optical signal pass state in the amorphous state. The input channels of the optical neural network can be the input channels corresponding to the cross-waveguide array. For example, the phase change material can correspond to a weight of 1 in the amorphous state and a weight of 0 in the crystalline state.

[0181] This application sets a weight calibration process based on the material properties of the phase change material to accurately set the weights of all storage and computing units in the cross-waveguide array. In specific implementation, each phase change material in the optical neural network can be set to a state where no voltage is applied, thereby initializing each phase change material to a full optical signal pass state. Then, a calibration light signal of the same intensity can be input to each input channel of the optical neural network.

[0182] For example, you can Figure 4 Each phase change material in the optical neural network shown is initialized to a full-pass state for optical signals, and calibrated optical signals of the same intensity are input to the nine input channels of the cross-waveguide array 1 of the optical neural network.

[0183] Step B: Collect the optical power output by each output channel corresponding to the cross-waveguide array in the optical neural network, and calibrate the minimum optical power value among the optical powers as a preset value.

[0184] Here, when the optical neural network includes one cross waveguide array, only the optical power output by each output channel corresponding to the unique cross waveguide array can be collected; when the optical neural network includes multiple cross waveguide arrays, the optical power output by each output channel corresponding to each cross waveguide array can be collected separately, and the optical power output by each output channel corresponding to each cross waveguide array can be used to perform weight calibration on the phase change materials in each cross waveguide array according to steps C and D below. Alternatively, when the optical neural network includes multiple cross waveguide arrays, the present application can also perform weight calibration on the phase change materials in each cross waveguide array separately. For example, for the first cross waveguide array in the optical neural network, each phase change material in the cross waveguide array can be initialized to a full-pass state of the optical signal, and a calibration optical signal of the same intensity can be input to each input channel of the cross waveguide array, and then the optical power output by each output channel of the cross waveguide array can be collected, and the optical power can be used to perform weight calibration on the phase change materials in the cross waveguide array according to steps C and D below. After the weight calibration of the phase change materials in the first cross-waveguide array is completed, each phase change material in the second cross-waveguide array can be initialized to the optical signal full-pass state, and a calibration optical signal of the same intensity is input to each input channel of the second cross-waveguide array. Then, the optical power output by each output channel of the second cross-waveguide array is collected, and the optical power is used to perform weight calibration on the phase change materials in the second cross-waveguide array according to steps C and D below. After the weight calibration of the phase change materials in the second cross-waveguide array is completed, the above steps are repeated to perform weight calibration on the phase change materials in the third cross-waveguide array until the weight calibration of the phase change materials in each cross-waveguide array in the optical neural network is completed.

[0185] In a specific implementation, for example, weight calibration of the phase change materials in each cross-waveguide array in an optical neural network can be performed simultaneously. The optical power outputted by each output channel corresponding to each cross-waveguide array can be collected, and the minimum optical power value among these optical powers can be calibrated as a preset value. The preset value can be, for example, 1.

[0186] Step C: Determine the target optical power of each phase change material according to the preset value, the minimum optical power, and the convolution kernel weight corresponding to each phase change material.

[0187] In specific implementation, the target optical power of each phase change material under a convolution kernel weight that needs to be characterized can be inferred based on the corresponding relationship between the convolution kernel weights obtained after network training and the preset value and the minimum optical power.

[0188] Step D: According to the target optical power of each phase change material, heat each phase change material separately so that its optical constant value matches the convolution kernel weight.

[0189] In specific implementation, each phase change material can be heated in turn to the target light power according to the target light power of each phase change material, so that the optical constant value of each phase change material is consistent with the convolution kernel it needs to characterize, thereby realizing the weight calibration of each phase change material.

[0190] For example, the calibration process for phase change materials can be as follows: all phase change materials are placed in a no-voltage state (all-pass optical signal state), a calibration optical signal of the same intensity is input to each input port of the optical neural network in turn, and the optical power of each output port of the optical neural network is recorded, with the minimum optical power value calibrated to 1. Based on the convolution kernel weight of the model, the corresponding optical power is calculated, and each phase change material is calibrated in turn so that its output optical power is consistent with the expected value. Since electricity is required when calibrating the weight of the phase change material, and no electricity is required for subsequent use of the phase change material once the calibration is completed, near-zero power reasoning can be achieved.

[0191] In another embodiment, the software-level calibration is specifically reflected in the network training process. Specifically, the convolution kernel weights can be obtained by training through the following steps:

[0192] During the iterative training of the optical neural network, the convolution kernel weights quantized with a set bit value are used in the forward propagation process, and the convolution kernel weights are obtained by gradient calculation using floating-point numbers in the back propagation process.

[0193] Here, the set size bit value can be a set low bit value. Since the accuracy of the front-end optical neural network will not be particularly high (that is, operations are performed at low precision), such as 4 bits or 8 bits, quantized perception training is required, and the set size bit value can be, for example, 4 bits or 8 bits.

[0194] Thus, at the software level, this application primarily uses quantization-aware training to mitigate the loss of computational accuracy caused by the transition of model weights from high to low precision. Quantization-aware training integrates quantization into the training process, using low-bit quantized convolution kernel weights and activation values ​​during forward propagation while still using floating-point numbers for gradient calculations during backward propagation. This allows the model to gradually adapt to quantization errors during training, reducing accuracy loss.

[0195] To facilitate understanding of the embodiments of the present application, the following will take the data to be processed as an example of a picture to be processed, combined with Figure 4 The optical neural network structure of this application illustrates the data processing flow:

[0196] The image to be processed is traversed in 3×3 data blocks. For the sub-data currently traversed, the initial digital domain signal corresponding to the sub-data can be encoded into a first optical domain signal. The first optical domain signal includes sub-signals with nine wavelengths (i.e., λ1 to λ9). The nine sub-signals are input into each row of the cross-waveguide array 1. The nine storage-computational units corresponding to each row perform convolution processing on the nine sub-signals, respectively, to obtain the convolved signal corresponding to each row. The convolved signals of each row constitute the convolution result of convolution layer 1. Photoelectric conversion is performed using the photodetectors connected to each row of the cross-waveguide array 1 to obtain the first current signal corresponding to each row. The first current signal in each row is input to the microring modulator and MZM in the corresponding row of nonlinearity 1. A preset optical domain signal (i.e., nonlinearity 1 corresponds to a preset optical domain signal with wavelengths λ1 to λ9) is input to the MZM in each row of nonlinearity 1 using an independent laser source. The phase of the MZM and the refractive index of the microring modulator are adjusted using the first current signal. Based on the adjusted phase and refractive index, the input preset optical domain signal is nonlinearly modulated to produce a nonlinear result. Specifically, nonlinear result 1 is obtained corresponding to the preset optical domain signal with wavelength λ1, nonlinear result 2 is obtained corresponding to the preset optical domain signal with wavelength λ2, and so on, and nonlinear result 9 is obtained corresponding to the preset optical domain signal with wavelength λ9. Each of the nine nonlinear results is input to a single column in the crossed waveguide array 2. The nine integrated storage and computation units corresponding to that column perform convolution processing on the nonlinear results to produce the target convolved signal. Specifically, nonlinear results 1 to 9 are input to the nine integrated storage and computation units corresponding to the single column in the crossed waveguide array 2 for convolution processing to produce the target convolved signal. A photodetector connected to the cross-waveguide array 2 is used to perform photoelectric conversion on the target convolved signal to obtain a second current signal. The second current signal is input to the micro-ring modulator and MZM in the nonlinear 2, and an independent laser source is used to input a preset optical domain signal to the MZM in the nonlinear 2. The phase of the MZM and the refractive index of the micro-ring modulator are adjusted by the second current signal. Based on the adjusted phase and refractive index, the input preset optical domain signal is nonlinearly modulated to obtain a second optical domain signal corresponding to the sub-data. The second optical domain signal corresponding to the sub-data is converted into the target digital domain signal, that is, Figure 4 The target digital domain signal of a pixel point in the optical neural network processing result is obtained by continuously traversing the image to be processed and inputting the sub-data obtained in each traversal into the optical neural network for the convolution and nonlinear processing described above. The target digital domain signal of a pixel point corresponding to each sub-data in the image to be processed in the optical neural network processing result is obtained. The target digital domain signal of each sub-data is composed of Figure 4 The optical neural network processing results in . The optical neural network processing results are input into the classification neural network to obtain the classification results of the image to be processed.

[0197] Corresponding to the aforementioned embodiments of the data processing method, the present application also provides embodiments of a data processing device.

[0198] The embodiment of the data processing device of the present application can be applied to edge-side terminals. The device embodiment can be implemented through software, hardware, or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the terminal where it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, if Figure 5 As shown, this is a hardware structure diagram of the terminal where the data processing device 600 of this application is located. Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, the terminal where the device is located in the embodiment may also include other hardware according to the actual function of the terminal, which will not be described in detail.

[0199] Please refer to Figure 6 , is a schematic diagram of a data processing device provided in an embodiment of the present application, comprising:

[0200] A traversal module 601 is configured to traverse the data to be processed according to a convolution calculation rule, and for the currently traversed sub-data, encode the initial digital domain signal corresponding to the sub-data into a first optical domain signal;

[0201] Processing module 602 is configured to input the first optical domain signal into an optical neural network for at least one convolution process and nonlinear processing to obtain a second optical domain signal of the sub-data on an output channel; convolution kernel weights used in the convolution process are characterized by optical constant values ​​of phase change materials in each integrated storage and computing unit provided in the optical neural network;

[0202] The classification module 603 is used to convert the second optical domain signals corresponding to the sub-data obtained through the traversal into target digital domain signals and input them into the classification neural network to obtain the classification result of the data to be processed.

[0203] In one possible implementation, the optical neural network includes at least one hidden layer, the hidden layer including a cross-waveguide array for convolution processing and a micro-ring modulation array for nonlinear processing; the cross-waveguide array includes a plurality of integrated storage and computing units;

[0204] The processing module 602, when inputting the first light domain signal into the optical neural network for at least one convolution process and nonlinear process to obtain the second light domain signal of the sub-data on the output channel, is configured to:

[0205] Inputting the first light domain signal into a first hidden layer in the optical neural network, and performing convolution processing on the first light domain signal using each storage-computation integrated unit in the first hidden layer to obtain a convolution result;

[0206] Inputting the convolution result into the micro-ring modulation array of the first hidden layer for nonlinear processing to obtain a nonlinear result;

[0207] In the case where a next hidden layer exists, inputting the nonlinear result into the next hidden layer to obtain a nonlinear result output by the next hidden layer;

[0208] The nonlinear result output by the last hidden layer is used as the second optical domain signal of the sub-data on the output channel.

[0209] In one possible implementation, the storage-computing integrated unit includes at least input and output waveguides, a directional coupler, and a phase change material;

[0210] The multiple storage-computing integrated units are arranged according to a set structure, each storage-computing integrated unit in a row or a column of the arrangement corresponds to a convolution kernel, and each convolution kernel weight of the convolution kernel is characterized by an optical constant value of a phase change material of each storage-computing integrated unit;

[0211] The processing module 602, when performing convolution processing on the first optical domain signal using the storage-computation integrated units in the first hidden layer to obtain a convolution result, is configured to:

[0212] For any convolution kernel of the first hidden layer, input the first optical domain signal to each storage-computing integrated unit corresponding to the convolution kernel through each input waveguide corresponding to the convolution kernel;

[0213] For any integrated storage and computing unit, the optical energy of the first optical domain signal at a set ratio is coupled using a directional coupler in the integrated storage and computing unit, and a convolution kernel weight represented by an optical constant value of the phase change material is used for multiplication processing, and the multiplication result of the integrated storage and computing unit is obtained and output through an output waveguide;

[0214] Determining an initial convolution result corresponding to the first optical domain signal under the convolution kernel according to a multiplication result of output waveguides of the respective storage-computation integrated units corresponding to the convolution kernel;

[0215] The convolution result of the first light domain signal is determined according to the initial convolution results corresponding to the first light domain signal under each of the convolution kernels.

[0216] In a possible implementation, the micro-ring modulation array includes a plurality of micro-ring modulators, wherein a doped region is provided in the micro-ring modulator, and the doped region includes P-type semiconductor carriers and N-type semiconductor carriers;

[0217] The processing module 602, when inputting the convolution result into the micro-ring modulation array of the first hidden layer for nonlinear processing to obtain a nonlinear result, is configured to:

[0218] Converting the convolution result into a current signal and inputting the current signal into each micro-ring modulator in the micro-ring modulation array of the first hidden layer;

[0219] Using the current signal to drive the carrier concentration in the doped region to change, thereby changing the refractive index of the micro-ring modulator, and performing nonlinear modulation based on the changed refractive index to obtain an initial modulation result;

[0220] The nonlinear result is obtained based on the initial modulation results corresponding to each micro-ring modulator.

[0221] In a possible implementation, the micro-ring modulation array further includes a first preset modulator cascaded with each micro-ring modulator;

[0222] The processing module 602 is configured to:

[0223] Inputting the current signal into the first preset modulator of each micro-ring modulator cascade, and adjusting the phase of the first preset modulator;

[0224] Using an independent laser source provided for each first preset modulator, a preset optical domain signal is inputted into each first preset modulator respectively;

[0225] The preset optical domain signal is nonlinearly modulated using the adjusted phase and the changed refractive index to obtain the initial modulation result.

[0226] In a possible implementation manner, the traversal module 601, when encoding the initial digital domain signal corresponding to the sub-data into the first optical domain signal, is configured to:

[0227] converting an initial digital domain signal corresponding to the sub-data into an analog domain signal, and inputting the analog domain signal into each second preset modulator;

[0228] Performing multi-wavelength modulation on the analog domain signal using each of the second preset modulators to obtain a modulated signal corresponding to each of the second preset modulators;

[0229] Using the wavelength division multiplexer associated with each second preset modulator, the modulated signal corresponding to each second preset modulator is separated and copied according to the signal wavelength to obtain a demultiplexed signal;

[0230] A delay line is used to perform dislocation processing on the demultiplexed signal to obtain the first optical domain signal.

[0231] In a possible implementation, the apparatus further includes a first calibration module 604, configured to calibrate the weight of the phase change material representation through the following steps:

[0232] Initializing each phase change material in the optical neural network to a light signal all-pass state, and inputting a calibration light signal of the same intensity into each input channel of the optical neural network;

[0233] Collecting the optical power output by each output channel corresponding to the cross-waveguide array in the optical neural network, and calibrating the minimum optical power value among the optical powers as a preset value;

[0234] Determining the target optical power of each phase change material according to the preset value, the minimum optical power, and the convolution kernel weight corresponding to each phase change material;

[0235] According to the target light power of each phase change material, each phase change material is heated respectively so that its optical constant value matches the convolution kernel weight.

[0236] In a possible implementation, the apparatus further includes a second calibration module 605, configured to obtain the convolution kernel weights through training in the following manner:

[0237] During the iterative training of the optical neural network, the convolution kernel weights quantized by the set bit value are used in the forward propagation process, and the convolution kernel weights are obtained by gradient calculation using floating point numbers in the backward propagation process.

[0238] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0239] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0240] Embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0241] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform the corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0242] Computers suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to such mass storage devices to receive data from them or to transmit data to them, or both. However, a computer does not necessarily have such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0243] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0244] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of specific embodiments of specific inventions. Certain features described in multiple embodiments within this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of the sub-combination.

[0245] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0246] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential sequence to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0247] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Traversing the data to be processed according to the convolution calculation rule, and encoding the initial digital domain signal corresponding to the currently traversed sub-data into a first optical domain signal; Inputting the first optical domain signal into an optical neural network for at least one convolution process and nonlinear processing to obtain a second optical domain signal of the sub-data on an output channel; a convolution kernel weight used in the convolution process is characterized by an optical constant value of a phase change material in each integrated storage and computing unit provided in the optical neural network; The second optical domain signals corresponding to the respective sub-data obtained through the traversal are converted into target digital domain signals and input into the classification neural network to obtain the classification results of the data to be processed; the method further includes the step of calibrating the convolution kernel weights characterized by the phase change material: Initializing each phase change material in the optical neural network to a light signal all-pass state, and inputting a calibration light signal of the same intensity into each input channel of the optical neural network; Collecting the optical power output by each output channel corresponding to the cross-waveguide array in the optical neural network, and calibrating the minimum optical power value among the optical powers as a preset value; Determining the target optical power of each phase change material according to the preset value, the minimum optical power, and the convolution kernel weight corresponding to each phase change material; According to the target light power of each phase change material, each phase change material is heated respectively so that its optical constant value matches the convolution kernel weight.

2. The method according to claim 1, characterized in that The optical neural network includes at least one hidden layer, the hidden layer includes a cross-waveguide array for convolution processing and a micro-ring modulation array for nonlinear processing; the cross-waveguide array includes a plurality of storage and computing integrated units; Inputting the first light domain signal into an optical neural network for at least one convolution process and nonlinear process to obtain a second light domain signal of the sub-data on an output channel includes: Inputting the first light domain signal into a first hidden layer in the optical neural network, and performing convolution processing on the first light domain signal using each storage-computation integrated unit in the first hidden layer to obtain a convolution result; Inputting the convolution result into the micro-ring modulation array of the first hidden layer for nonlinear processing to obtain a nonlinear result; In the case where a next hidden layer exists, inputting the nonlinear result into the next hidden layer to obtain a nonlinear result output by the next hidden layer; The nonlinear result output by the last hidden layer is used as the second optical domain signal of the sub-data on the output channel.

3. The method according to claim 2, characterized in that The storage and computing integrated unit at least includes input and output waveguides, a directional coupler and a phase change material; The multiple storage-computing integrated units are arranged according to a set structure, each storage-computing integrated unit in a row or a column of the arrangement corresponds to a convolution kernel, and each convolution kernel weight of the convolution kernel is characterized by an optical constant value of a phase change material of each storage-computing integrated unit; The step of performing convolution processing on the first optical domain signal using each storage-computation integrated unit in the first hidden layer to obtain a convolution result includes: For any convolution kernel of the first hidden layer, input the first optical domain signal to each storage-computing integrated unit corresponding to the convolution kernel through each input waveguide corresponding to the convolution kernel; For any integrated storage and computing unit, the optical energy of the first optical domain signal at a set ratio is coupled using a directional coupler in the integrated storage and computing unit, and a convolution kernel weight represented by an optical constant value of the phase change material is used for multiplication processing, and the multiplication result of the integrated storage and computing unit is obtained and output through an output waveguide; Determining an initial convolution result corresponding to the first optical domain signal under the convolution kernel according to a multiplication result of output waveguides of the respective storage-computation integrated units corresponding to the convolution kernel; The convolution result of the first light domain signal is determined according to the initial convolution results corresponding to the first light domain signal under each of the convolution kernels.

4. The method according to claim 2, characterized in that The micro-ring modulation array includes a plurality of micro-ring modulators, wherein a doped region is provided in the micro-ring modulator, and the doped region includes P-type semiconductor carriers and N-type semiconductor carriers; Inputting the convolution result into the micro-ring modulation array of the first hidden layer for nonlinear processing to obtain a nonlinear result includes: Converting the convolution result into a current signal and inputting the current signal into each micro-ring modulator in the micro-ring modulation array of the first hidden layer; Using the current signal to drive the carrier concentration in the doped region to change, thereby changing the refractive index of the micro-ring modulator, and performing nonlinear modulation based on the changed refractive index to obtain an initial modulation result; The nonlinear result is obtained based on the initial modulation results corresponding to each micro-ring modulator.

5. The method according to claim 4, characterized in that The micro-ring modulation array further includes a first preset modulator cascaded with each micro-ring modulator; The performing nonlinear modulation based on the changed refractive index to obtain an initial modulation result includes: Inputting the current signal into the first preset modulator of each micro-ring modulator cascade, and adjusting the phase of the first preset modulator; Using an independent laser source provided for each first preset modulator, a preset optical domain signal is inputted into each first preset modulator respectively; The preset optical domain signal is nonlinearly modulated using the adjusted phase and the changed refractive index to obtain the initial modulation result.

6. The method according to claim 1, characterized in that The encoding of the initial digital domain signal corresponding to the sub-data into a first optical domain signal includes: converting an initial digital domain signal corresponding to the sub-data into an analog domain signal, and inputting the analog domain signal into each second preset modulator; Performing multi-wavelength modulation on the analog domain signal using each of the second preset modulators to obtain a modulated signal corresponding to each of the second preset modulators; Using the wavelength division multiplexer associated with each second preset modulator, the modulated signal corresponding to each second preset modulator is separated and copied according to the signal wavelength to obtain a demultiplexed signal; A delay line is used to perform dislocation processing on the demultiplexed signal to obtain the first optical domain signal.

7. The method according to claim 1, characterized in that The convolution kernel weights are trained as follows: During the iterative training of the optical neural network, the convolution kernel weights quantized by the set bit value are used in the forward propagation process, and the convolution kernel weights are obtained by gradient calculation using floating point numbers in the backward propagation process.

8. A data processing device, characterized in that: The device comprises: a traversal module, configured to traverse the data to be processed according to a convolution calculation rule, and for the currently traversed sub-data, encode an initial digital domain signal corresponding to the sub-data into a first optical domain signal; a processing module, configured to input the first optical domain signal into an optical neural network for at least one convolution process and nonlinear processing to obtain a second optical domain signal of the sub-data on an output channel; a convolution kernel weight used in the convolution process being characterized by an optical constant value of a phase change material in each integrated storage and computing unit provided in the optical neural network; The classification module is configured to convert the second optical domain signals corresponding to the respective sub-data obtained through the traversal into target digital domain signals and input them into the classification neural network to obtain the classification results of the data to be processed. The device also includes a first calibration module, configured to calibrate the convolution kernel weights representing the phase change material through the following steps: Initializing each phase change material in the optical neural network to a light signal all-pass state, and inputting a calibration light signal of the same intensity into each input channel of the optical neural network; Collecting the optical power output by each output channel corresponding to the cross-waveguide array in the optical neural network, and calibrating the minimum optical power value among the optical powers as a preset value; Determining the target optical power of each phase change material according to the preset value, the minimum optical power, and the convolution kernel weight corresponding to each phase change material; According to the target light power of each phase change material, each phase change material is heated respectively so that its optical constant value matches the convolution kernel weight.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images