Data processing method and device, storage medium and computer equipment

Abstract

The invention provides a data processing method and device, a storage medium and computer equipment, and the method comprises the steps: carrying out the traversal of to-be-processed data according to a convolution calculation rule, and encoding an initial digital domain signal corresponding to the subdata into a first optical domain signal for the current traversal subdata; inputting the first optical domain signal into an optical neural network for convolution processing and nonlinear processing to obtain a second optical domain signal of the sub-data; the convolution kernel weight used in the convolution processing is represented by using the optical constant value of the phase change material in each storage and calculation integrated unit set in the optical neural network; and converting the second optical domain signal corresponding to each subdata obtained by traversing into a target digital domain signal, and inputting the target digital domain signal into a classification neural network to obtain a classification result. The method breaks through the limitation of an existing optical neural network in the aspects of power consumption and depth, has the advantages of high efficiency, low delay, expandability and the like, and is suitable for edge intelligent calculation scenes of tasks such as image classification.

Description

Technical Field

[0001] The present application relates to the field of optical computing, and more specifically, to a data processing method, apparatus, storage medium, and computer device. Background Art

[0002] Edge Computing, as a distributed computing framework, is initiated from the edge side of the network, aiming to generate faster network service responses, thereby reducing the latency problem caused by data transmission. However, due to the limited computing power of the edge devices required for edge computing, there is a relatively high latency in neural network model inference tasks.

[0003] With the development of optical computing, by virtue of its unique characteristics of large bandwidth and high parallelism, replacing electronic computing with optical computing can bring the effects of reducing response latency and increasing throughput. Therefore, if optical computing can be applied to the neural network models used in edge computing, it is expected to reduce the latency and resource consumption of edge computing.

[0004] However, currently, when applying an optical neural network computing chip for edge computing, electricity is still used to control network weights, and the huge number of weights will still cause problems such as inference delay and high computing power consumption in the optical neural network, with obvious drawbacks. Summary of the Invention

[0005] In view of this, the present application provides a data processing method, apparatus, storage medium, and computer device to reduce the inference latency and computing power consumption when an optical neural network is applied to edge computing.

[0006] Specifically, the present application is implemented through the following technical solutions: In a first aspect, an embodiment of the present disclosure provides a data processing method, including: Traverse the data to be processed according to the rules of convolutional calculation. For the currently traversed sub-data, encode the initial digital domain signal corresponding to the sub-data into a first optical domain signal; Input the first optical domain signal into an optical neural network for at least one convolutional process and non-linear process to obtain a second optical domain signal of the sub-data on the output channel; the convolutional kernel weights used in the convolutional process are characterized by the optical constant values of phase change materials in each memory-computation integrated unit set in the optical neural network; Convert the second optical domain signals respectively corresponding to the traversed sub-data into target digital domain signals and input them into a classification neural network to obtain the classification result of the data to be processed.

[0007] In a possible implementation, the optical neural network includes at least one hidden layer, and the hidden layer includes a cross-waveguide array for convolutional processing and a microring modulation array for non-linear processing; the cross-waveguide array includes a plurality of memory-computation integrated units; Inputting the first optical-domain signal into the optical neural network for at least one convolutional processing and non-linear processing to obtain a second optical-domain signal of the sub-data on the output channel includes: Input the first optical-domain signal into the first hidden layer in the optical neural network, and use each memory-computation integrated unit in the first hidden layer to perform convolutional processing on the first optical-domain signal to obtain a convolutional result; Input the convolutional result into the microring modulation array of the first hidden layer for non-linear processing to obtain a non-linear result; In the case of the existence of a next hidden layer, input the non-linear result into the next hidden layer to obtain a non-linear result output by the next hidden layer; Use the non-linear result output by the last hidden layer as the second optical-domain signal of the sub-data on the output channel.

[0008] In a possible implementation, the memory-computation integrated unit at least includes input / output waveguides, directional couplers, and phase change materials; The plurality of memory-computation integrated units are arranged according to a set structure. Each memory-computation integrated unit in a row or a column under the arrangement corresponds to a convolution kernel, and each convolution kernel weight of the convolution kernel is respectively characterized by the optical constant value of the phase change material of each memory-computation integrated unit; Using each memory-computation integrated unit in the first hidden layer to perform convolutional processing on the first optical-domain signal to obtain a convolutional result includes: For any convolution kernel in the first hidden layer, input the first optical-domain signal into each memory-computation integrated unit corresponding to the convolution kernel through the respective input waveguides corresponding to the convolution kernel; For any memory-computation integrated unit, use the directional coupler in the memory-computation integrated unit to couple the optical energy of the first optical-domain signal at a set ratio, and perform multiplication processing using the convolution kernel weight characterized by the optical constant value of the phase change material to obtain a multiplication result of the memory-computation integrated unit and output it through the output waveguide; Determine an initial convolutional result of the first optical-domain signal corresponding to the convolution kernel according to the multiplication results output by the output waveguides of each memory-computation integrated unit corresponding to the convolution kernel; Determine the convolutional result of the first optical-domain signal according to the initial convolutional results of the first optical-domain signal corresponding to each convolution kernel.

[0009] In a possible implementation, the microring modulation array includes a plurality of microring modulators, a doped region is provided in the microring modulator, and the doped region includes P-type semiconductor carriers and N-type semiconductor carriers; Inputting the convolution result into the microring modulation array of the first hidden layer for non-linear processing to obtain a non-linear result includes: Converting the convolution result into a current signal and inputting it into each microring modulator in the microring modulation array of the first hidden layer; Using the current signal to drive the change in the carrier concentration in the doped region, changing the refractive index of the microring modulator, and performing non-linear modulation based on the changed refractive index to obtain an initial modulation result; Based on the initial modulation results corresponding to each microring modulator, obtaining the non-linear result.

[0010] In a possible implementation, the microring modulation array further includes a first preset modulator cascaded with each microring modulator; The performing non-linear modulation based on the changed refractive index to obtain an initial modulation result includes: Inputting the current signal into the first preset modulator cascaded with each microring modulator to adjust the phase of the first preset modulator; Using an independent laser source provided for each first preset modulator to input a preset optical domain signal into each first preset modulator respectively; Using the adjusted phase and the changed refractive index to perform non-linear modulation on the preset optical domain signal to obtain the initial modulation result.

[0011] In a possible implementation, the encoding the initial digital domain signal corresponding to the sub-data into a first optical domain signal includes: Converting the initial digital domain signal corresponding to the sub-data into an analog domain signal and inputting the analog domain signal into each second preset modulator; Using each second preset modulator to perform multi-wavelength modulation on the analog domain signal to obtain a modulation signal corresponding to each second preset modulator; Using a wavelength division demultiplexer associated with each second preset modulator to separate and replicate the modulation signal corresponding to each second preset modulator according to the signal wavelength to obtain a demultiplexed signal; Using a delay line to perform dislocation processing on the demultiplexed signal to obtain the first optical domain signal.

[0012] In a possible implementation, the method further includes a step of calibrating the weights characterized by the phase change material: Initialize each phase change material in the optical neural network to the all-pass state of the optical signal, and input calibration optical signals of the same intensity into each input channel of the optical neural network; Collect the optical powers output from the respective output channels corresponding to the cross-waveguide array in the optical neural network, and calibrate the value of the minimum optical power among the respective optical powers to a preset value; Determine the target optical power of each phase change material according to the preset value, the minimum optical power, and the convolution kernel weight corresponding to each phase change material; Heat each phase change material according to the target optical power of each phase change material so that its optical constant value matches the convolution kernel weight.

[0013] In a possible implementation manner, the convolution kernel weight is trained through the following method: During the iterative training of the optical neural network, during the forward propagation process, the convolution kernel weight quantized by a set-size bit value is used, and during the backward propagation process, floating-point numbers are used for gradient calculation to obtain the convolution kernel weight.

[0014] In a second aspect, an embodiment of the present disclosure further provides a data processing device, including: A traversal module, configured to traverse the data to be processed according to the rules of convolution calculation, and encode the initial digital domain signal corresponding to the currently traversed sub-data into a first optical domain signal; A processing module, configured to input the first optical domain signal into the optical neural network for at least one convolution process and non-linear process to obtain a second optical domain signal of the sub-data on the output channel; the convolution kernel weight used in the convolution process is characterized by the optical constant value of the phase change material in each memory and computing unit set in the optical neural network; A classification module, configured to convert the second optical domain signals corresponding to the respective sub-data obtained by traversal into target digital domain signals and input them into a classification neural network to obtain a classification result of the data to be processed.

[0015] In a third aspect, an optional implementation manner of the present disclosure further provides a computer-readable storage medium, including a computer program, and when the computer program is run by a processor, the steps in the first aspect or any possible implementation manner in the first aspect are implemented.

[0016] In a fourth aspect, an optional implementation manner of the present disclosure further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the steps in the first aspect or any possible implementation manner in the first aspect are implemented.

[0017] The data processing method, apparatus, storage medium, and computer device provided by the embodiments of the present disclosure can stably store convolution kernel weights in a non-powered state since the optical constant value of the phase change material is fixed once. Therefore, by implementing convolution calculations in the memory-computation integrated unit of the optical neural network and using the optical constant value of the phase change material deployed in the memory-computation integrated unit to represent the convolution kernel weights, convolution calculations can be performed using weights with nearly zero static power consumption, thereby reducing the computational power consumption and inference latency of the optical neural network. When using the optical neural network to perform convolution calculations on each sub-data obtained by traversal, there is no need to electrically control the weights each time when using the weights, which expands the computing ability of the phase change material to the entire network layer level, thereby greatly improving the inference speed of the data to be processed and reducing the inference power consumption. Finally, by using the classification neural network to perform classification processing in edge computing on the second optical domain signals corresponding to each sub-data, accurate classification results can be obtained. The in-memory computing architecture based on phase change materials proposed in this application (i.e., each memory-computation integrated unit in the cross-waveguide array) can achieve multi-layer neural network inference with nearly zero power consumption, effectively improving the inference response speed and reducing the computational power consumption.

[0018] To make the above objects, features, and advantages of the present disclosure more obvious and understandable, the following specifically gives preferred embodiments and detailed descriptions in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of a data processing method shown in an exemplary embodiment of the present application; Figure 2 is a schematic structural diagram of a data encoding module shown in an exemplary embodiment of the present application; Figure 3 is a specific implementation flowchart of a data processing method shown in an exemplary embodiment of the present application; Figure 4 is a schematic structural diagram of an optical neural network shown in an exemplary embodiment of the present application; Figure 5 is a hardware structure diagram of a terminal where a data processing apparatus 600 is located shown in an exemplary embodiment of the present application; Figure 6 is a schematic diagram of a data processing apparatus shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0021] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "the", and "said" used in this application and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0022] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0023] It has been found through research that edge computing, as a distributed computing framework, moves the tasks of data processing and analysis from a centralized data center to edge devices on the network side, thereby greatly reducing the latency problem caused by data transmission. However, edge devices usually do not have very strong computing power, and have a relatively high response latency and large computing energy consumption in the task of electrical neural network inference. Although a method of applying an optical neural network to edge computing has been proposed currently, the conventional application method still uses electricity to control the size of neural network elements (i.e., uses electricity to control the size of convolution kernel weights). The large number of neurons results in the optical neural network still having a relatively high static power consumption and being difficult to meet the requirements of edge computing. Therefore, how to reduce the computing latency and computing power consumption of the optical neural network when applying the optical neural network to edge computing has become a technical problem worthy of attention.

[0024] Based on the above research, the present disclosure provides a data processing method, apparatus, storage medium, and computer device. Since once the optical constant value of the phase change material is fixed, the convolution kernel weights can be stabilized in a non-powered state, convolution calculation can be implemented in the memory-computation integrated unit of the optical neural network, and the optical constant value of the phase change material deployed in the memory-computation integrated unit can be used to represent the convolution kernel weights, thereby enabling convolution calculation using weights with near-zero static power consumption, reducing the computational power consumption and inference latency of the optical neural network. When using the optical neural network to perform convolution calculations on each of the traversed sub-data respectively, there is no need to electrically control the weights each time when using the weights, which realizes the extension of the computing ability of the phase change material to the entire network layer level, thus greatly improving the inference speed of the data to be processed and reducing the inference power consumption. Finally, using the classification neural network to perform classification processing in edge computing on the second optical domain signals corresponding to each sub-data, accurate classification results can be obtained. The in-memory computing architecture based on phase change materials proposed in this application can achieve multi-layer neural network inference with near-zero power consumption, effectively improving the inference response speed and reducing the computational power consumption.

[0025] Regarding the defects existing in the above solutions, they are all the results obtained by the inventors through practice and careful research. Therefore, the process of discovering the above problems and the solutions proposed by the present disclosure for the above problems in the following text both belong to the contributions made by the inventors to the present disclosure during the process of the present disclosure.

[0026] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0027] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to users and the authorization of users should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0028] It should be noted that the specific terms mentioned in the embodiments of the present disclosure include: U-Net: A convolutional neural network architecture for image segmentation, designed to handle biomedical image segmentation tasks; P-type semiconductor: Also known as hole-type semiconductor, it is a semiconductor that conducts electricity mainly through positively charged holes; N-type semiconductor: Also known as electron-type semiconductor, that is, an impurity semiconductor in which the free electron concentration is much greater than the hole concentration.

[0029] For ease of understanding this embodiment, first, a data processing method disclosed in this disclosure embodiment will be introduced in detail. The execution subject of the data processing method provided in this disclosure embodiment is generally a terminal device or other processing device with certain computing capabilities. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant device (PDA), a handheld device, a computer device, etc.; in some possible implementation manners, the data processing method may be implemented by a processor invoking computer-readable instructions stored in a memory.

[0030] Next, the data processing method provided in this disclosure embodiment will be described by taking the execution subject as an edge-side device as an example.

[0031] As Figure 1 shown, it is a flowchart of a data processing method provided in this disclosure embodiment, which may include the following steps: S101: Traverse the data to be processed according to the rules of convolution calculation. For the currently traversed sub-data, encode the initial digital domain signal corresponding to the sub-data into a first optical domain signal.

[0032] Here, the data processing method provided in this application embodiment can be applied to edge computing tasks in an edge-side device, such as an image classification task, a text processing task, etc.

[0033] The optical neural network can be a pre-trained neural network based on optical computing. The rules of convolution calculation are used to indicate information such as the size of the sliding window and the sliding step during convolution processing. Taking the sliding window size of 3×3 as an example, the data to be processed can be traversed according to the data block size of 3×3 and the sliding step to obtain each sub-data.

[0034] The data to be processed is related to the edge computing task specifically applied by the optical neural network. For example, when the edge computing task is an image classification task, the data to be processed can be image data to be processed, video data to be processed, etc.; when the edge computing task is a text processing task, the data to be processed can be text to be processed, document to be processed, etc.

[0035] The sub-data can be each data block obtained by each traversal. For example, when traversing the image to be processed, the data to be processed can be each image data block obtained by traversal.

[0036] The initial digital domain signal is the digital signal of the sub-data in the digital domain, and the first optical domain signal is the optical domain signal in the optical domain. Since the data to be processed is usually a signal in the data domain, while the input of the optical neural network needs to be an optical domain signal, it is necessary to convert the initial digital domain signal of the sub-data into an optical domain signal to input the optical neural network for data processing. Taking the data to be processed as the picture to be processed as an example, the data corresponding to each pixel point in the image is stored as a digital signal, that is, a digital domain signal.

[0037] Exemplarily, to implement the convolution operation of the optical neural network on the data to be processed, it is necessary to design a corresponding data encoding module for the optical neural network to encode the data to be processed. Among them, the data encoding module includes a data encoding process and a corresponding hardware link. Specifically, after obtaining the data to be processed, the size of the data block required for each traversal can be determined according to the rules of convolution calculation, and then the data to be processed can be traversed according to the size of the data block to obtain the sub-data blocks obtained from each traversal, and the data corresponding to the sub-data blocks is used as the sub-data. During the traversal process, for the currently traversed sub-data, the initial digital domain signal corresponding to the sub-data can be encoded into the first optical domain signal through a pre-designed data encoding process and a corresponding hardware link.

[0038] Optionally, the digital domain-optical domain conversion technology in the prior art can also be used to encode the initial digital domain signal of the currently traversed sub-data into the first optical domain signal.

[0039] In one embodiment, the data encoding module in the present application may include a plurality of second preset modulators and a wavelength division demultiplexer (abbreviated as DMUX) associated with each second preset modulator. Among them, the second preset modulator is a modulator for optically modulating an analog domain signal. Exemplarily, the second preset modulator may be a Mach-Zehnder Modulator (abbreviated as MZM). Among them, the number of second preset modulators is related to the number of input channels of the cross-waveguide array and the convolution kernel size. For example, when the number of input channels of the cross-waveguide array is 9, the convolution kernel size is 3×3, and there are 3 wavelengths for the second preset modulator, the number of second preset modulators can be 3; when the number of input channels is 16, the convolution kernel size is 4×4, and there are 4 wavelengths for the second preset modulator, the number of second preset modulators can be 4. The following will describe the step of "encoding the initial digital domain signal corresponding to the sub-data into the first optical domain signal" in S101 above in combination with the specific structure of the data encoding module: S101-1: Convert the initial digital domain signal corresponding to the sub-data into an analog domain signal, and input the analog domain signal into each second preset modulator.

[0040] Here, the analog-domain signal can be an analog signal in the electrical analog domain, and the second preset modulator can be an MZM.

[0041] During specific implementation, when traversing the data to be processed according to the rules of convolution calculation, slicing of the data to be processed can be achieved, and each slice is the sub-data corresponding to the sub-data block obtained by each traversal. Then, each row in the sub-data obtained by each traversal can be separately input into an MZM in the data encoding module for electro-optic modulation.

[0042] Taking the data block with a size of 3×3 as the sub-data and the MZM including 3 as an example, the analog-domain signal of the first row data of the 3×3 size data block can be input into the first MZM for electro-optic modulation; the analog-domain signal of the second row data can be input into the second MZM for electro-optic modulation; the analog-domain signal of the third row data can be input into the third MZM for electro-optic modulation.

[0043] S101-2: Use each second preset modulator to perform multi-wavelength modulation on the analog-domain signal to obtain the modulation signals corresponding to each second preset modulator.

[0044] Here, the number of wavelengths in the multi-wavelength modulation can be related to the number of input channels of the cross-waveguide array and / or the convolution kernel size and / or the network design structure. For example, when the number of input channels of the cross-waveguide array is 9 and the convolution kernel size is 3×3, the number of wavelengths can be 3; when the number of input channels of the cross-waveguide array is 16 and the convolution kernel size is 4×4, the number of wavelengths can be 4. After the second preset modulator performs multi-wavelength modulation on the input analog-domain signal, a modulation signal including multiple wavelengths can be output. Since each row of the analog-domain signal corresponding to a sub-data will be input into different second preset modulators for electro-optic modulation, each second preset modulator can output a modulation signal related to the sub-data.

[0045] During specific implementation, after inputting the analog-domain signal corresponding to the sub-data into each second modulator, for each second preset modulator, the second preset modulator can perform multi-wavelength modulation on the signal intensity of the input analog-domain signal to obtain a modulation signal including multiple wavelengths.

[0046] S101-3: Use the wavelength demultiplexer associated with each second preset modulator to separate and replicate the modulation signal corresponding to each second preset modulator according to the signal wavelength to obtain the demultiplexed signal.

[0047] In specific implementation, a wavelength division multiplexer is associated with each second preset modulator, and the modulated signals output by each second preset modulator can be input into the associated wavelength division multiplexer. The wavelength division multiplexer separates and duplicates the modulated signals according to the signal wavelength and the number of wavelengths of the modulated signals, obtaining multiple demultiplexed signals.

[0048] For example, if the modulated signal includes 3 wavelengths, after separating and duplicating using the wavelength division multiplexer, 3 demultiplexed signals can be obtained, and the optical intensities of the three demultiplexed signals are the same but the wavelengths are different.

[0049] S101-4: Use a delay line to perform misalignment processing on the demultiplexed signal to obtain a first optical domain signal.

[0050] In specific implementation, for each demultiplexed signal, a delay line can be used to perform delay processing on the demultiplexed signal, so that the demultiplexed signals with different wavelengths are misaligned in time, obtaining a misaligned signal. The obtained misaligned signal is used as the first optical domain signal and input into the optical neural network for processing.

[0051] Exemplarily, after the modulated signal output by a certain second preset modulator is processed by the wavelength division multiplexer and the delay line, multiple misaligned signals with different wavelengths but the same intensity can be obtained.

[0052] In this way, the data encoding process of this application utilizes the translation invariance property of convolution, reduces the number of second preset modulators required, and reduces the power consumption of data conversion. For example, when the number of input channels of the cross-waveguide array is 9 and the size of the convolution kernel is 3×3, if the data encoding process of this application is not used, 9 second preset modulators are required for electro-optical modulation, while this application only needs to use 3 second preset modulators, effectively reducing the number of second preset modulators used and reducing the conversion power consumption.

[0053] As Figure 2 shown, it is a schematic structural diagram of a data encoding module provided by an embodiment of this application. Among them, in Figure 2 it is described by taking the number of input channels of the cross-waveguide array as 9 and the size of the convolution kernel as 3×3 as an example. Among the data to be processed, ~ represent the initial digital domain signals corresponding to the pixel points in the first row and first column to the pixel points in the m-th row and n-th column in the data to be processed; the values of m and n are related to the size of the data to be processed; Figure 2 only shows the initial digital domain signals corresponding to some pixel points, and the initial digital domain signals corresponding to the remaining pixel points are represented by ellipsis. After traversing the data to be processed, each slice can be obtained, that is, each sub-data can be obtained. In Figure 2Only three sub-data obtained after three traversals are shown, but the number of sub-data obtained in the actual application process is determined according to the size of the data to be processed. Figure 2 Only taking three sub-data (sub-data 1 to 3) as an example, each sub-data includes 9 pixel points. Figure 2 In each of the sub-data represents the intensity of the analog-domain signal corresponding to the pixel point in the sub-data, where the value of a is less than or equal to m, and the value of b is less than or equal to n. For example, Figure 2 in sub-data 1 of represents the pixel point in the sub-data corresponding to the intensity of the analog-domain signal. represents the pixel point in the sub-data corresponding to the intensity of the analog-domain signal. For sub-data 1, three rows of data can be respectively input into three Mach-Zehnder modulators (i.e., MZM-1 to MZM-3) for multi-wavelength modulation to obtain three modulation signals. Then, the modulation signals output by MZM-1 to MZM-3 are respectively input into the corresponding associated demultiplexer (DMUX) for optical signal separation and replication and form time-misaligned signals through delay lines to obtain nine misaligned signals, and the nine misaligned signals are used as the first optical-domain signal corresponding to sub-data 1 and input into the optical neural network for processing. For example, for Figure 2 the nine misaligned signals in Figure 2 in the fifth column of ), can be multiplied and accumulated with the nine weights of the convolution kernel in the optical neural network respectively to achieve the convolution processing of one convolution kernel. In Figure 2 MAM-1 corresponds to three misaligned signals, that is, three signals with different wavelengths ; MAM-2 corresponds to three misaligned signals, that is, three signals with different wavelengths ; MAM-3 corresponds to three misaligned signals, that is, three signals with different wavelengths . In Figure 2 shows a cross-waveguide array of the optical neural network for convolution processing, where each row represents a convolution kernel, and each square in each row represents 1 storage and computing unit corresponding to the convolution kernel.

[0054] In this way, by using the translational invariance characteristic of convolution, in Figure 2 nine misaligned signals can be obtained by using 3 MZMs instead of 9 MZMs, and convolution calculations can be respectively performed with the nine weights of the convolution kernel, effectively reducing the power consumption of data conversion.

[0055] It should be noted that in this application, after the sub-data obtained from each traversal is inferred using the optical neural network to obtain the second optical domain signal corresponding to the sub-data, the sub-data obtained from the next traversal will be inferred using the optical neural network. For example, in Figure 2 after traversing to obtain sub-data 1, encoding the initial digital domain signal corresponding to sub-data 1 into the first optical domain signal and using the optical neural network to infer the second optical domain signal of sub-data 1, then the next traversal will be performed to obtain sub-data 2, encoding the initial digital domain signal corresponding to sub-data 2 into the first optical domain signal and using the optical neural network to infer the second optical domain signal of sub-data 2, and then the next traversal will be performed to obtain sub-data 3, …, until the second optical domain signal corresponding to the last sub-data is obtained.

[0056] S102: Input the first optical domain signal into the optical neural network for at least one convolution process and non-linear process to obtain the second optical domain signal of the sub-data on the output channel; the convolution kernel weights used in the convolution process are characterized by the optical constant values of the phase change materials in each memory-computation integrated unit set in the optical neural network.

[0057] Here, the optical neural network can be entirely deployed in the optical computing chip and can accelerate the feature extraction process of the first optical domain signal to obtain the second optical domain signal corresponding to the sub-data. The above output channel can be the output channel of the last network layer of the optical neural network. The convolution kernel weights can be the respective network weights corresponding to the convolution kernel, and the data of the network weights corresponding to a convolution kernel can be determined according to the network structure of the convolution layer and / or convolution requirements. For example, when the convolution kernel size is 3×3, the convolution kernel corresponds to 9 network weights. The optical constant value of the phase change material can specifically be the crystallization state of the phase change material.

[0058] In view of the problem of high static power consumption existing in the existing optical neural network, the present invention proposes a memory-in-computation architecture based on phase change materials, which can achieve near-zero power consumption for inference of each layer of the neural network. Specifically, the in-memory computing structure can include at least one convolution layer (i.e., the cross-waveguide array described later) and a non-linear layer (i.e., the microring modulation array described later) connected to each convolution layer. Among them, the number of convolution layers can be determined according to the task requirements and scale of the optical neural network, and no specific limitation is made in the embodiments of this application. The first optical domain signal can sequentially pass through each convolution layer and non-linear layer for convolution processing and non-linear processing respectively to obtain the second optical domain signal corresponding to the sub-data and complete feature extraction.

[0059] Since the phase change material is non-volatile and is in a crystalline state or an amorphous state after different thermal drives are applied, its optical constant value changes with the state, which is used to simulate the weight value. Once the optical constant value of the phase change material is determined by thermal drive, the optical constant value will not change even in the non-powered state. Therefore, once the optical constant value of the phase change material is set, the corresponding network weight can be maintained without continuous power supply during subsequent use. Among them, the network weight is the convolution kernel weight of the optical neural network. Once the optical constant value of the phase change material is fixed, the optical constant value of the phase change material can still be used to characterize the convolution kernel weight even in the non-powered state.

[0060] Multiple memory-computation integrated units can be set in the optical neural network. Each memory-computation integrated unit is used for one weight multiplication operation. The number of memory-computation integrated units is related to the convolution kernel size and the number of convolution kernels. For example, when the convolution kernel size is 3×3 and the number of convolution kernels is 9, the number of memory-computation integrated units can be 81. The memory-computation integrated unit includes a phase change material, and the optical constant value of the phase change material is used to characterize a weight of a convolution kernel.

[0061] During specific implementation, the sub-data corresponding to each first optical domain signal matching the number of input channels of the cross-waveguide array can be input into the optical neural network. Each memory-computation integrated unit in the optical neural network is used to perform a convolution operation on each first optical domain signal, and then the nonlinear layer is used to perform nonlinear processing on the convolution result to obtain the second optical domain signal of the sub-data on the output channel. Among them, the weight multiplication operation in the convolution operation can be simulated by the phase change material attenuating the energy of the first optical domain signal. For example, the convolution kernel weight characterized by the optical constant value of the phase change material in the memory-computation integrated unit can be used to perform a multiplication operation on the first optical domain signal.

[0062] In this way, the optical neural network is used to perform convolution and nonlinear processing on the first optical domain signal corresponding to the sub-data obtained by each traversal, so as to realize the feature extraction of the first optical domain signal and obtain the second optical domain signal corresponding to the sub-data obtained by each traversal.

[0063] This application takes into account that phase change materials are a type of non-volatile storage material, which can retain neural network weights in an optical neural network without power supply. Phase change materials can switch back and forth between the crystalline state and the amorphous state, and different states will cause significant changes in the optical constant values, thereby being able to represent the size of the convolution kernel weights in the neural network. Introducing phase change materials into the optical neural network can effectively reduce the high static power consumption problem of optical computing chips, and is expected to meet the energy consumption requirements of edge-side computing. The research work on using phase change materials for optical computing in the prior art is in its infancy, and the technical feasibility of using phase change materials to achieve optical computing has been demonstrated at the level of individual operators such as matrix multiplication and convolution. This application extends the computing ability of phase change materials at the operator level to the entire network level (for example, for an N-layer optical neural network, all can be implemented using phase change materials deployed on an optical computing chip), designs an optical deep neural network computing architecture, and realizes near-zero power consumption optical neural network inference. For ease of understanding, the prior art can only achieve the calculation of a single operator, such as matrix multiplication; while the network level is equivalent to constructing an optical neural network using phase change materials and using phase change materials to achieve the convolution calculation of the optical neural network. In this application, the convolution calculation of multiple hidden layers is achieved using phase change materials, so the application of phase change materials at the network level is realized.

[0064] S103: Convert the second optical domain signals respectively corresponding to the traversed sub-data into target digital domain signals and input them into the classification neural network to obtain the classification result of the data to be processed.

[0065] Here, the classification neural network can be a trained electrical neural network for performing classification tasks. The input of the classification neural network needs to be a digital domain signal. Therefore, after obtaining the second optical domain signals corresponding to each sub-data, the second optical domain signals need to be converted into digital domain signals so that the classification neural network can process them. The classification result is used to indicate the classification result of the data to be processed under the classification task in edge computing.

[0066] In specific implementation, the second optical domain signals respectively corresponding to each sub-data can be converted into electrical signals through a photoelectric detector (abbreviated as PD), and then the electrical signals are encoded into target digital domain signals, and then the target digital domain signals corresponding to each sub-data are input into the classification neural network to obtain the classification result of the data to be processed. Among them, the process of encoding the electrical signal into the target digital domain signal can be to first encode the electrical signal into an analog domain signal, and then convert the analog domain signal into the target digital domain signal. In this way, for the second optical domain signal output from the output end of the optical neural network, photoelectric conversion and analog-to-digital conversion are sequentially performed to obtain the target digital domain signal that the classification neural network can process. Optionally, the classification neural network can be a classification network in the prior art.

[0067] For the optoelectronic hybrid neural network proposed in this application, the data to be processed is modulated into the optical domain through electro-optic conversion, and the optical domain signal undergoes high-speed and low-power feature extraction through an optical neural network. The output signal of the optical neural network is reconverted into the electrical digital domain and then the data classification task is completed through the subsequent classification neural network.

[0068] As Figure 3 shown, the following is a specific implementation flowchart of a data processing method provided by this application, which may include the following steps: The data to be processed ( Figure 3 taking the image to be processed as an example) is modulated into the optical domain through electro-optic conversion and input into the front-end optical neural network for feature extraction to obtain an optical domain signal. The optical domain signal is input into the subsequent classification neural network for classification processing to obtain a classification result. In Figure 3 , the classification result is the 9th among 9 results. In this way, through the optoelectronic hybrid neural network computing architecture proposed in this application, the inference latency and computing power consumption of the network can be reduced, thereby meeting the requirements of edge-side computing performance. Specifically, the optoelectronic hybrid network is divided into two parts: the front end and the back end. The front-end network performs feature extraction on the data to be processed, which is implemented through an optical neural network, and the back-end network performs feature classification, which is implemented through a classification neural network in the electrical digital domain.

[0069] For the network classification task in edge computing, this application divides the network into two parts: the front-end optical neural network and the back-end classification neural network. The front-end network completes feature extraction of the data, and the back-end network classifies according to the features extracted by the front-end network. The in-memory computing optical computing chip designed in this application can be used for data processing in the front-end network to accelerate the feature extraction process and reduce the computing power consumption of this part. The overall data processing flow is as follows: (1) Extract the data in the image in the electrical domain in sequence; (2) Convert the electrical domain digital signal into an optical domain analog signal through digital-to-analog conversion and electro-optic conversion; (3) The optical domain analog signal completes the convolution operation and nonlinear processing of the image through the optical neural network to obtain a modulated optical signal; (4) Convert the modulated optical signal output by the optical neural network into an electrical domain digital signal through optoelectronic conversion and analog-to-digital conversion; (5) The electrical domain digital signal completes task classification through the subsequent classification neural network.

[0070] In one embodiment, the optical neural network may include at least one hidden layer, where each hidden layer includes a cross-waveguide array for convolutional processing and a microring modulation array for non-linear processing; wherein, the microring modulation array is connected after the cross-waveguide array, the cross-waveguide array includes a plurality of memory-computation integrated units, and one cross-waveguide array is equivalent to a convolutional layer, which can implement convolutional processing of data and complete feature extraction of the data to be processed; one microring modulation array is equivalent to a non-linear layer, which can implement non-linear processing of data; that is, the convolutional layer is implemented through the network architecture of the cross-waveguide array, and the non-linear layer is implemented through the microring modulation array. The structure of the microring modulation array is related to the structure of the cross-waveguide array. For example, for each row of the cross-waveguide array, there is a corresponding row in the microring modulation array. This application supports building multiple hidden layers in the optical neural network, thereby increasing the depth of the optical neural network.

[0071] The number of hidden layers is related to the task requirements and network scale of the optical neural network. Two adjacent hidden layers in each hidden layer are connected, and the output of the previous hidden layer can be used as the input of the next hidden layer. The output of the last hidden layer is the second optical domain signal. The convolutional kernel sizes corresponding to the cross-waveguide arrays in different hidden layers can be different. Each cross-waveguide array in each hidden layer has a set number of input channels and a set number of output channels. The number of input channels and output channels of the cross-waveguide arrays in different hidden layers can be different, or the number of input channels of the previous cross-waveguide array can be the same as the number of output channels of the next cross-waveguide array, but the number of output channels of the next cross-waveguide array is different from the number of output channels of the previous cross-waveguide array. In this way, different hidden layers can be used to expand or compress the number of channels of the feature map of the data to be processed.

[0072] One hidden layer can also be referred to as a network layer, which is used to perform one convolutional processing and one non-linear processing on the data to be processed. Therefore, the number of convolutional processing times and non-linear processing times performed by the optical neural network on the data to be processed is related to the number of hidden layers in the optical neural network.

[0073] For example, an optical neural network can implement the data processing flow in convolutional neural networks such as U-Net in the prior art, perform convolution and non-linear processing on the data to be processed, and obtain a processing result. Specifically, the optical neural network can include two hidden layers. The cross-waveguide array in hidden layer 1 can be used to expand the number of channels of the data to be processed. After expansion, convolution with different convolution kernels in the cross-waveguide array can obtain diverse convolution features, thereby improving the diversity of convolution features. The cross-waveguide array in hidden layer 2 can be used to compress the number of channels of the data to be processed, so that the optical neural network can eliminate redundant feature information and reduce the optoelectronic conversion pressure and analog-to-digital conversion pressure at the output end of the optical neural network.

[0074] For the above S102, it can be implemented according to the following steps: S102-1: Input the first optical domain signal into the first hidden layer of the optical neural network, and use each memory-computation integrated unit in the first hidden layer to perform convolution processing on the first optical domain signal to obtain a convolution result.

[0075] In specific implementation, after obtaining the first optical domain signal of the sub-data, the first optical domain signal can be input into the cross-waveguide array in the first hidden layer of the optical neural network, and each memory-computation integrated unit included in the cross-waveguide array in the first hidden layer is used to perform convolution processing on the first optical domain signal (i.e., complete the dot product operation of the first optical domain signal and the convolution kernel weight), thereby realizing feature extraction of the first optical domain signal and obtaining a convolution result.

[0076] S102-2: Input the convolution result into the microring modulation array of the first hidden layer for non-linear processing to obtain a non-linear result.

[0077] In specific implementation, the convolution result output by the cross-waveguide array in the first hidden layer can be input into the microring modulation array in the first hidden layer for non-linear processing to obtain a non-linear result. Among them, the non-linear result is a signal in the optical domain.

[0078] For example, the non-linear processing of the present application can adopt an opto-electro-optic implementation method. Specifically, the convolution result can be converted into a current signal through a photodetector, and the carrier concentration of the microring modulator in the microring modulation array is driven by the current signal, thereby realizing modulation of the resonance curve. The properties of the resonance curve determine the non-linear response relationship between the modulation electrical signal and the output optical signal, thereby obtaining a non-linear result.

[0079] S102-3: In the case where there is a next hidden layer, input the non-linear result into the next hidden layer to obtain a new non-linear result output by the next hidden layer.

[0080] In the case where the optical neural network includes multiple hidden layers, each hidden layer processes the data to be processed sequentially. Therefore, after obtaining the non-linear result using the microring modulation array of the previous hidden layer, if there is a next hidden layer, the non-linear result output by the previous hidden layer can be input to the next hidden layer for convolution processing and non-linear processing to obtain a new non-linear result. If there is no next hidden layer, the non-linear result output by the current hidden layer can be used as the second optical domain signal of the sub-data on the output channel.

[0081] Exemplarily, after the first hidden layer finishes processing, if there is a next hidden layer, the non-linear result obtained by processing the first hidden layer is input to the next hidden layer for processing to obtain the non-linear result output by the next hidden layer. Among them, the processing process of the next hidden layer is also to first perform convolution processing on the non-linear result output by the previous hidden layer using the cross-waveguide array in this hidden layer, and then perform non-linear processing on the convolution processing result using the microring modulation array in this hidden layer to obtain the new non-linear result output by this hidden layer. Among them, the array sizes of the cross-waveguide arrays in different hidden layers can be different, and the array sizes of the microring modulation arrays can also be different.

[0082] S102-4: Use the non-linear result output by the last hidden layer as the second optical domain signal of the sub-data on the output channel.

[0083] Specifically, whenever a non-linear result output by a hidden layer is obtained, if there is a next hidden layer, the non-linear result can be input to the next hidden layer for processing. If not, it means that the current hidden layer is the last hidden layer, and the non-linear result output by the last hidden layer can be used as the second optical domain signal of the sub-data on the output channel. Among them, the number of output channels of the cross-waveguide array in the last hidden layer is the number of output channels at the output end of the optical neural network.

[0084] As Figure 4 shown, it is a schematic structural diagram of an optical neural network provided by an embodiment of the present application. The processing process of the optical neural network will be described below in combination with Figure 4 the content: Taking Figure 4 the optical neural network in which includes 2 hidden layers as an example, the first hidden layer can include convolution layer 1 (i.e., cross-waveguide array 1) and non-linearity 1 (i.e., microring modulation array 1), and the second hidden layer can include convolution layer 2 (i.e., cross-waveguide array 2) and non-linearity 2 (i.e., microring modulation array 2). The second hidden layer is located after the first hidden layer. Both non-linearity 1 and non-linearity 2 are used for non-linear processing, that is, for performing Processing. In convolutional layer 1, the dimension of the cross-waveguide array is 9×9, which can simultaneously perform convolution operations with 9 different convolutional kernels. The size of the convolutional kernel is 3×3. The number of input channels of the cross-waveguide array 1 is 9, and the number of output channels is 9. Figure 4 K1~K9 in convolutional layer 1 are the 9 convolutional kernels of convolutional layer 1. Each row of the cross-waveguide array represents the convolutional kernel weights corresponding to a convolutional kernel. Since the number of rows of the cross-waveguide array 1 is 9, convolution operations with 9 different convolutional kernels can be performed in parallel. After obtaining the data to be processed (i.e., Figure 4 the picture to be processed shown), the picture to be processed can be traversed first according to the data block size of 3×3. Each traversal obtains a sub-data block of 3×3 size. The initial digital domain signal corresponding to the sub-data block is encoded into a first optical domain signal. In this way, a first optical domain signal composed of optical domain signals corresponding to 9 pixel points in the sub-data block can be obtained. As Figure 4 λ1~λ9 below convolutional layer 1 in are the wavelengths of the optical domain signals corresponding to the 9 pixel points respectively. The first optical domain signal completes the dot product operation between the small block data and the convolutional kernel weights through the cross-waveguide array 1 to obtain a convolution result, and then the convolution result is input to Nonlinear 1 for nonlinear processing, and the nonlinear result after processing by the first hidden layer can be obtained. Figure 4 The dimension of convolutional layer 2 in is 9×1. There is one convolutional kernel in convolutional layer 2, and the size of the convolutional kernel is 1×1. The number of input channels of the cross-waveguide array 2 is 9, and the number of output channels is 1. The nonlinear result after processing by the first hidden layer is input to the cross-waveguide array 2 for convolution processing to obtain a convolution processing result, and the result is input to Nonlinear 2 for nonlinear processing to obtain a nonlinear result, and the nonlinear result is used as the second optical domain signal of the sub-data on the output channel.

[0085] In one embodiment, each memory-computation unit may at least include an input-output waveguide, a directional coupler, and a phase change material. The input-output waveguide may be a cross waveguide constructed using silicon nitride SiN. Specifically, the waveguide may include an input waveguide and an output waveguide arranged in a cross shape. For example, in Figure 4Among them, the vertical waveguide in the memory - in - computing unit is used to indicate the input waveguide, and the horizontal waveguide is used to represent the output waveguide. The directional coupler can be constructed using silicon (Si). The state of the phase - change material is controlled by an externally applied voltage. The externally applied voltage can heat the heater around the phase - change material, and the heat is then conducted to the phase - change material, thereby controlling the crystallization or amorphous state of the phase - change material, and thus realizing the modification of the optical constant value of the phase - change material. When constructing an optical neural network, according to the convolution kernel weights of each convolution kernel, the optical constant value of the phase - change material in each memory - in - computing unit can be set by applying a voltage, so as to realize the representation of the convolution kernel weights. When applying the constructed optical neural network, the phase - change material can attenuate the energy of the input optical signal to simulate the weight multiplication operation in the neural network.

[0086] The multiple memory - in - computing units included in the cross - waveguide array can be arranged according to a set structure. Each memory - in - computing unit in a row or a column under the arrangement corresponds to a convolution kernel, and each convolution kernel weight of the convolution kernel is respectively represented by the optical constant value of the phase - change material of each memory - in - computing unit. Here, the set structure can be a cross - cross structure. Under the cross - cross structure, each row arranged can correspond to a convolution kernel, that is, each memory - in - computing unit in each row corresponds to a convolution kernel, and each convolution kernel weight of the convolution kernel is respectively represented by the optical constant value of the phase - change material of each memory - in - computing unit, or each memory - in - computing unit in each column corresponds to a convolution kernel, and each convolution kernel weight of the convolution kernel is respectively represented by the optical constant value of the phase - change material of each memory - in - computing unit. Regarding whether a row or a column under the set structure arrangement corresponds to a convolution kernel, it can be designed according to the specific network structure, and the embodiments of the present application do not make specific limitations. For example, in Figure 4 Among them, the dimension of the cross - waveguide array 1 is 9×9, there are a total of 9 convolution kernels, and the convolution kernel size is 3×3, that is Figure 4 The cross - waveguide array 1 in is a 9 - row × 9 - column array. Each row in this array corresponds to a convolution kernel, and the optical constant values of the phase - change materials in the 9 memory - in - computing units in each row are used to represent the 9 convolution kernel weights of the convolution kernel corresponding to that row. The cross - waveguide array structure in the convolutional layer 2 is the same as that in the convolutional layer 1, with a dimension of 9×1 and a convolution kernel size of 1×1. The number of input channels of the cross - waveguide array 2 is 9, and the number of output channels is 1, that is Figure 4 The cross - waveguide array 2 in is a 9 - row × 1 - column array. Each column in this array corresponds to a convolution kernel, and the optical constant values of the phase - change materials in the 9 memory - in - computing units in each column are used to represent the 9 convolution kernel weights of the convolution kernel corresponding to that column.

[0087] For the above S102 - 1, it can be implemented according to the following steps: S102-1-1: For any convolution kernel in the first hidden layer, input the first optical domain signal into each computing-in-memory unit corresponding to the convolution kernel through the respective input waveguides corresponding to the convolution kernel.

[0088] In specific implementation, the first optical domain signal corresponding to the sub-data can be input into each row of the cross waveguide array in the first hidden layer. For each row, the first optical domain signal can be input into each computing-in-memory unit included in the row through the input waveguides in each computing-in-memory unit included in the row.

[0089] For example, in Figure 4 , the first optical domain signal of the 3×3 size sub-data block of the picture to be processed (i.e., the optical domain signals corresponding to λ1~λ9 below the first convolutional layer) can be respectively input into the 9 computing-in-memory units included in each row of the cross waveguide array 1 through the input waveguides in the computing-in-memory units in each row of the cross waveguide array 1. For example, using Figure 4 , the input waveguides of the 9 computing-in-memory units in the first row of the cross waveguide array 1 in Figure 4 , the optical domain signals corresponding to λ1~λ9 of the first optical domain signal are respectively input into the 9 computing-in-memory units in the first row; using Figure 4 , the input waveguides of the 9 computing-in-memory units in the second row of the cross waveguide array 1 in Figure 4 , the optical domain signals corresponding to λ1~λ9 of the first optical domain signal are respectively input into the 9 computing-in-memory units in the second row; and so on, it can be realized that the optical domain signals corresponding to λ1~λ9 of the first optical domain signal are respectively input into the computing-in-memory units corresponding to the 9 rows of the cross waveguide array 1.

[0090] S102-1-2: For any computing-in-memory unit, use the directional coupler in the computing-in-memory unit to couple the optical energy of the first optical domain signal at a set ratio, and perform multiplication processing using the convolution kernel weight characterized by the optical constant value of the phase change material, and obtain the multiplication result of the computing-in-memory unit and output it through the output waveguide.

[0091] Here, the set ratio can be the ratio under the total energy of the input optical domain signal, and its size can be related to the number of convolution kernels. For example, the set ratio can be 1 / number of convolution kernels. The reason for coupling the optical energy of the first optical halo signal at a set ratio is to enable the optical domain signals of each input channel to be evenly distributed to multiple computing-in-memory units through the directional coupler array to achieve signal replication. Among them, the directional coupler array is the array composed of the directional couplers included in each computing-in-memory unit in the cross waveguide array. For example, in Figure 4In it, the optical domain signal corresponding to λ1 in the cross-waveguide array 1 can be evenly distributed in each memory-computation integrated unit corresponding to the first column through the directional coupler array. The optical domain signal corresponding to λ2 in the cross-waveguide array 1 can be evenly distributed in each memory-computation integrated unit corresponding to the second column through the directional coupler array.

[0092] During specific implementation, for any memory-computation integrated unit corresponding to any convolution kernel, the directional coupler in this memory-computation integrated unit can be used to couple the optical energy of the first optical domain signal at a set ratio, and the convolution kernel weight characterized by the optical constant value of the phase change material in this memory-computation integrated unit is used to attenuate the optical energy coupled by the directional coupler, so as to implement the weighted multiplication operation on the first optical domain signal, obtain the multiplication result and output it through the output waveguide of this memory-computation integrated unit. For example, in Figure 4 In it, for the memory-computation integrated unit in the first row and first column of the cross-waveguide array 1, the directional coupler in this memory-computation integrated unit can be used to couple 1 / 9 of the optical energy of the optical domain signal corresponding to λ1, and the convolution kernel weight characterized by the optical constant value of the phase change material in this memory-computation integrated unit is used to perform weighted multiplication processing on the optical energy coupled by the directional coupler, obtain the multiplication result and output it using the output waveguide in this memory-computation integrated unit, and obtain the multiplication result of λ1 under the convolution kernel corresponding to the first row.

[0093] S102-1-3: Determine the initial convolution result of the first optical domain signal under the convolution kernel according to the multiplication results output by the output waveguides of each memory-computation integrated unit corresponding to the convolution kernel.

[0094] During specific implementation, for any convolution kernel corresponding to any cross-waveguide array, the output channel corresponding to this convolution kernel can be used to perform an accumulation calculation on the multiplication results output by the output waveguides of each memory-computation integrated unit corresponding to this convolution kernel, and obtain the initial convolution result of the first optical domain signal under this convolution kernel.

[0095] For example, in Figure 4 In it, for the convolution kernel corresponding to the first row in the cross-waveguide array 1, this convolution kernel can correspond to an output channel, and the multiplication results output by the output waveguides of the memory-computation integrated units in the first row are accumulated to obtain the initial convolution result of the first optical domain signal under the convolution kernel corresponding to the first row.

[0096] In this way, the input end of the cross-waveguide array includes multiple input channels to load a multi-dimensional vector, for example Figure 4The 9 input channels of the cross-waveguide array 1 in the image are used to load the first optical domain signal corresponding to the 9 pixel points in the sub-data block. The first optical domain signal is evenly distributed to each storage-computation integrated unit through a directional coupler to complete the signal replication. The replicated signal is subjected to energy attenuation according to the convolution kernel weight represented by the phase change material to complete the multiplication operation of the signal and the convolution kernel weight. Finally, the output end of the cross-waveguide array includes multiple output channels, each of which collects the input signals modulated by the storage-computation integrated unit to complete the accumulation operation, that is, the output channel performs incoherent superposition of the input channel signals modulated by the storage-computation integrated unit to achieve the accumulation operation. Combining the above operations, the cross-waveguide array realizes the multiplication operation of the matrix (for example, the weight matrix corresponding to the 9×9 phase change material in the cross-waveguide array 1) and the vector. The present application utilizes the characteristics of local information processing and streaming computing of convolution computing. Each time, only a small piece of information in the data to be processed needs to be sent to the optical neural network for processing, without sending all the data to be processed into the network at one time, which reduces the requirements for the channel scale of the optical neural network.

[0097] S102-1-4: Determine the convolution result of the first light domain signal according to the initial convolution results corresponding to the first light domain signal under each convolution kernel.

[0098] During specific implementation, for the first hidden layer, the initial convolution results under each convolution kernel of the cross waveguide array 1 in the first hidden layer can be used as the convolution results of the first optical domain signal in the first hidden layer.

[0099] For example, in Figure 4 In the cross waveguide array 1, the initial convolution result of the first light domain signal under the convolution kernel corresponding to each row of the cross waveguide array can be obtained, that is, the initial convolution result of the first light domain signal under the convolution kernel corresponding to 9 rows can be obtained. Then, the initial convolution result under the convolution kernel corresponding to these 9 rows can be used as the convolution result of the first light domain signal under the cross waveguide array 1. Among them, the convolution result of the first light domain signal under the cross waveguide array 1 includes 9 convolution optical signals.

[0100] For the cross-waveguide array in any hidden layer in the optical neural network, its structure is similar to the structure of the cross-waveguide array in the first hidden layer introduced above, the difference lies in the number and size of convolution kernels and the number of input and output channels. Therefore, the process of convolution processing of optical domain signals by the cross-waveguide array in any hidden layer in the optical neural network can be similar to the process of convolution processing of optical domain signals by the cross-waveguide array in the first hidden layer introduced above, and this application will not repeat it here.

[0101] Still Figure 4For example, assume that the shape of the input image to be processed is C×D. After being processed by the cross-waveguide array 1, it will be transformed into three-dimensional cubic data with a data shape of C×D×9, where 9 is the number of convolution kernels in the cross-waveguide array 1. The array structure of the cross-waveguide array 2 is the same as that of the cross-waveguide array 1, with a dimension of 9×1, a convolution kernel size of 1×1, an input channel number of 9, and an output channel number of 1. The processing flow of the cross-waveguide array 2 can be as follows: Use the 9 input channels of the cross-waveguide array 2 to receive the optical signals respectively output by the 9 output channels of the hidden layer 1 (specifically, in Figure 4 it can specifically be the non-linear results of the 9 channels output by the non-linearity 1); Use the 9 memory-compute integrated units corresponding to one convolution kernel in the cross-waveguide array 2 to perform convolution processing to obtain the convolution result under one output channel. After passing through the cross-waveguide array 2, data compression can be performed in the channel dimension, that is, the number of channels of the feature map is compressed, so as to realize the compression of the data with a shape of C×D×9 output by the cross-waveguide array 1 into data with a shape of C×D. That is to say, after Figure 4 the convolution layer 1 in, 1 input image can be expanded into 9 images, and after passing through the convolution layer 2, the 9 images output by the convolution layer 1 can be compressed into 1 image.

[0102] In this application, the cross-waveguide array is used for convolution operation, and the convolution operation can be implemented by the method of matrix multiplying a vector. During convolution processing, the convolution kernel size is usually small. Therefore, in this application, there is no need to design a large-scale cross-waveguide array to store weight parameters (that is, there is no need to set a large number of memory-compute integrated units to use the optical constant values of the phase change materials in the memory-compute integrated units to represent the network weights), so it can adapt to the current computing scale of the optical neural network.

[0103] In one embodiment, the microring modulation array includes a plurality of microring modulators, and a doping region is provided in the microring modulator. The doping region includes P-type semiconductor carriers and N-type semiconductor carriers. Here, the number of microring modulators included in the microring modulation array in each hidden layer is related to the number of convolution kernels and / or output channels of the cross-waveguide array in that hidden layer. For example, in Figure 4 one, a microring modulator is connected after each convolution kernel corresponding to each row in the cross-waveguide array 1 (that is, when the cross-waveguide array 1 outputs nine data, there are nine microring modulators), and each microring modulator constitutes the microring modulation array of the non-linearity 1. A PN doping region is provided in each microring modulator. The P region in the PN doping region includes P-type semiconductor carriers, and the N region includes N-type semiconductor carriers.

[0104] For the above S102-2, it can be implemented according to the following steps: S102-2-1: Convert the convolution result into a current signal and input it into each microring modulator in the microring modulation array of the first hidden layer.

[0105] Here, the microring modulation array in the present application can implement non - linear calculation of signals, and the non - linear calculation adopts an opto - electro - optical implementation method.

[0106] During specific implementation, the convolution result output by the cross - waveguide array of the first hidden layer (this convolution result is an optical - domain signal) can be converted into a current signal through a photodetector; then the current signal is input into each microring modulator in the microring modulation array of the first hidden layer. For example, in Figure 4 , the 9 - way convolution optical signals output by the cross - waveguide array 1 can be respectively converted into electrical signals to obtain 9 - way current signals. For example, the convolution optical signals output by each row of the cross - waveguide array in the cross - waveguide array 1 can be converted into electrical signals through a photodetector and input into the microring modulator corresponding to this row in the non - linear 1.

[0107] S102 - 2 - 2: Use the current signal to drive the change in the carrier concentration in the doped region, change the refractive index of the microring modulator, and perform non - linear modulation based on the changed refractive index to obtain an initial modulation result.

[0108] During specific implementation, for each microring modulator, the voltage magnitude corresponding to the current signal input to this microring modulator can be used to drive the change in the carrier concentration in the PN - doped region on this microring modulator, thereby changing the material refractive index of this microring modulator, and further causing the microring resonance peak of this microring modulator to shift. Since the curve nature of the microring resonance peak determines the non - linear response relationship between the modulated electrical signal and the output optical signal, the change in the microring resonance peak will achieve non - linear modulation of the optical signal to obtain an initial modulation result. The modulated optical signal will be used as the input for the cross - waveguide array in the next hidden layer for the next - layer convolution calculation, thereby realizing multi - layer optical neural network inference.

[0109] For example, in Figure 4 , the 9 - way convolution optical signals output by the cross - waveguide array 1 have different light intensities detected by the photodetector, so the magnitudes of the 9 - way current signals obtained after conversion by the photodetector are also different. After being respectively converted into 9 - way current signals, each electrical signal can be respectively input into the corresponding microring modulator in the non - linear 1, and the voltage signal corresponding to the electrical signal is used to change the material refractive index of the microring modulator, thereby realizing the modulation of the electrical signal to obtain an initial modulation result. Since the magnitudes of the current signals input to different microring modulators in the non - linear 1 are different, the material refractive indices changed by different microring modulators are also different.

[0110] S102 - 2 - 3: Based on the initial modulation results corresponding to each microring modulator, obtain a non - linear result.

[0111] During specific implementation, the initial modulation results corresponding to each micro-ring modulator can be used together as the non-linear result output by the micro-ring modulation array.

[0112] For example, in Figure 4 , the initial modulation results corresponding to each micro-ring modulator in Nonlinear 1 can be used as the non-linear result output by the micro-ring modulation array corresponding to Nonlinear 1.

[0113] In the optical computing chip of this application, an optical non-linear processing unit (i.e., a micro-ring modulation array) is designed during the inference process of each layer of the optical neural network. This non-linear processing unit enables the optical signal to not require any digital-to-analog conversion process during the convolution inference process of the optical neural network, thereby reducing the computing power consumption and latency.

[0114] In one embodiment, in the non-linear processing part, in addition to setting micro-ring modulators, a first preset modulator can also be set. Specifically, the micro-ring modulation array can also include a first preset modulator cascaded with each micro-ring modulator. Here, the first preset modulator can be a Mach-Zehnder modulator MZM. The cascaded use of the MZ modulator and the micro-ring modulator can increase the complexity of the non-linear response, thereby enabling the optical neural network to have better performance. A micro-ring modulator and the first preset modulator cascaded with this micro-ring modulator can be defined as 1 unit in the micro-ring modulation array. For example, in Figure 4 , a Nonlinear 1 is further provided after the Crossed Waveguide Array 1, and a Nonlinear 2 is further provided after the Crossed Waveguide Array 2. Nonlinear 1 includes 9 units associated with the 9 output channels of the Crossed Waveguide Array 1, and each unit includes a micro-ring modulator and a first preset modulator. Nonlinear 2 includes 1 unit associated with the 1 output channel of the Crossed Waveguide Array 2, and this unit includes a micro-ring modulator and a first preset modulator.

[0115] Regarding the step of "performing non-linear modulation based on the changed refractive index to obtain the initial modulation result" in S102-2-2 above, it can also be implemented according to the following steps 1 to 3: Step 1: Input the current signal into the first preset modulator cascaded with each micro-ring modulator to adjust the phase of the first preset modulator.

[0116] During specific implementation, the convolution result output by the crossed waveguide array of the first hidden layer (this convolution result is an optical domain signal) can be converted into a current signal through a photodetector; then the current signal is input into each micro-ring modulator in the micro-ring modulation array of the first hidden layer, and the first preset modulator cascaded with each micro-ring modulator. While the current signal changes the refractive index of the micro-ring modulator, the current signal can also act on the first preset modulator, thereby changing the phase of the phase shifter of the first preset modulation.

[0117] Step 2: Using the independent laser sources set for each first preset modulator, input the preset optical domain signals into each first preset modulator respectively.

[0118] Here, the present application can equip each first preset modulator in the microring modulation array with an independent laser, that is, each unit in the microring position array is correspondingly provided with an independent laser. The input of the additional optical domain signal is realized through the independent lasers configured for each first preset modulator. The intensities of the optical domain signals output by the respective independent lasers are the same, but the wavelengths are different. For example, in Figure 4 , each MZM in Nonlinear 1 is provided with an independent laser, and each independent laser inputs an additional optical domain signal to the associated MZM. The wavelengths of the additional optical domain signals input by the respective independent lasers can be λ1~λ9 in Nonlinear 1. The additional optical domain signal here is the preset optical domain signal with a preset intensity.

[0119] In specific implementation, while inputting the respective current signals corresponding to the cross-waveguide array into the microring modulation array, using the independent laser sources set for each first preset modulator, input the preset optical domain signals into each first preset modulator respectively.

[0120] For example, in Figure 4 , the input of Nonlinear 1 includes not only the respective current signals corresponding to Cross-Waveguide Array 1, but also the preset optical domain signals input by the independent laser sources set for each first preset modulator. For example, λ1~λ9 in Nonlinear 1 respectively correspond to the wavelengths of different preset optical domain signals.

[0121] Step 3: Using the adjusted phase and the changed refractive index, perform nonlinear modulation on the preset optical domain signal to obtain an initial modulation result.

[0122] In specific implementation, after the current signal changes the refractive index of the microring modulator and the phase of the first preset modulator, the adjusted phase and the changed refractive index can be used to perform nonlinear modulation on the preset optical domain signal input to the first preset modulator to obtain an initial modulation result. For example, in Figure 4 , for each unit in Nonlinear 1, after using the current signal to change the refractive index of the microring modulator and the phase of the MZM in this unit, based on the adjusted phase and the changed refractive index, perform nonlinear modulation on the preset optical domain signal input to the MZM in this unit to obtain an initial modulation result. In this way, λ1~λ9 in Nonlinear 1 respectively correspond to the wavelengths of different preset optical domain signals. Although the intensities of the 9-channel preset optical domain signals are the same, after being modulated by the microring modulator and the MZM, 9-channel initial modulation results with different intensities will be obtained.

[0123] Exemplarily, Figure 4The 9 optical signals after convolution in the cross waveguide array 1 will be converted into 9 current signals, and the 9 current signals will act on the corresponding units respectively, resulting in the change of the material refractive index of the microring modulator in the unit and the change of the phase of the phase shifter of the MZM modulator (since different current signals correspond to different voltages, different voltages will lead to different refractive indices, and different refractive indices will lead to different phase changes of the phase shifter, so the degree of phase change of the phase shifter of each MZM modulator is different). The change of refractive index and the change of phase will affect the intensity of the preset optical domain signal input from the independent laser into the unit, so as to obtain the final modulated initial modulation result. For example, Figure 4 The preset optical domain signal corresponding to λ1 in the nonlinearity 1 will change in optical intensity after entering the MZM. This change is caused by the phase change of the phase shifter of the MZM, and the phase change is caused by the magnitude of the current signal input to the MZM. After the MZM modulates the optical intensity, the resonant curve of the microring modulator is used to modulate the preset optical domain signal with changed intensity again, so as to obtain the initial modulation result of the preset optical domain signal corresponding to λ1.

[0124] It can be understood that the structures of each nonlinear layer (i.e., each microring modulation array) in the optical neural network are similar to the structure of the nonlinear layer in the first hidden layer introduced above. The difference is that the number of units in the microring modulation array is different. Therefore, the nonlinear modulation processes of each nonlinear layer in the optical neural network are similar to the nonlinear modulation process of the nonlinear layer in the first hidden layer introduced above, which will not be elaborated here.

[0125] In this way, in the nonlinear processing part, each unit is provided with an independent laser to input an independent laser light source. On the one hand, it solves the problem of signal coherence between the output channels of the convolution layer in front of the nonlinear processing part, which leads to the problem that it cannot be cascaded with the next-level cross waveguide array. On the other hand, it makes the depth of the optical neural network not limited by the loss of the overall link, making it possible to build a deeper neural network structure. On the contrary, if no independent laser is set and the convolution layer 1 and the convolution layer 2 are directly cascaded together, since there is optical coherence between the 9 optical signals output by the convolution layer 1, and the 9 optical signals input to the convolution layer 2 require incoherence, it will cause the convolution layer 2 to be unable to process or the processing effect is not ideal. By setting the independent laser, the coherent optical signals output by the convolution layer 1 are adjusted to incoherent optical signals, thus meeting the input requirements of the convolution layer 2.

[0126] In this application, by designing the network structure of the cross waveguide array, the network weights do not need to be maintained by a static voltage, and the convolution processing of the optical neural network can be realized with near-zero power consumption. In addition, a microring modulation array is introduced into the optical neural network to realize the nonlinear calculation of signals, so as to complete the inference of a multi-layer deep neural network.

[0127] In one embodiment, the network weights of the optical neural network can be obtained through training. After the model training is completed, the network weights of the front-end optical neural network need to be deployed to the phase change materials in each memory-computation integrated unit in the optical computing chip. To improve the accuracy of weight deployment, the present invention proposes corresponding optimization strategies at the hardware level and the software level respectively to achieve the calibration of the network weights corresponding to the optical constant values of the phase change materials, and as much as possible alleviate the problem of the decline in network inference accuracy. Specifically, the calibration at the hardware level can be achieved through the following steps A to D: Step A: Initialize each phase change material in the optical neural network to the all-pass state of the optical signal, and input calibration optical signals of the same intensity to each input channel of the optical neural network.

[0128] Here, the all-pass state of the optical signal is the state of not absorbing the optical signal. The phase change material does not absorb light when in the amorphous state and absorbs all the light when in the crystalline state. Therefore, it can be known that the phase change material is in the all-pass state of the optical signal when in the amorphous state. Each input channel of the optical neural network can be each input channel corresponding to the cross-waveguide array. Exemplarily, the phase change material can correspond to weight 1 when in the amorphous state and weight 0 when in the crystalline state.

[0129] This application sets the weight calibration process according to the material characteristics of the phase change material to achieve accurate weight setting for all memory-computation integrated units in the cross-waveguide array. In specific implementation, each phase change material in the optical neural network can be set to the state of not applying voltage, so as to initialize each phase change material to the all-pass state of the optical signal. Then, calibration optical signals of the same intensity can be input to each input channel of the optical neural network.

[0130] For example, each phase change material in the Figure 4 shown optical neural network can be initialized to the all-pass state of the optical signal, and calibration optical signals of the same intensity are input to the 9 input channels of the cross-waveguide array 1 of the optical neural network.

[0131] Step B: Collect the optical powers output from each output channel corresponding to the cross-waveguide array in the optical neural network, and calibrate the value of the minimum optical power among the optical powers to a preset value.

[0132] Here, when an optical neural network includes a cross-waveguide array, the optical power output from each output channel corresponding to the only cross-waveguide array can be collected; when the optical neural network includes multiple cross-waveguide arrays, the optical power output from each output channel corresponding to each cross-waveguide array can be collected respectively, and the phase change materials in each cross-waveguide array can be weight-calibrated according to the optical power output from each output channel corresponding to each cross-waveguide array and steps C and D described later. Alternatively, when the optical neural network includes multiple cross-waveguide arrays, the present application can also weight-calibrate the phase change materials in each cross-waveguide array respectively. For example, for the first cross-waveguide array in the optical neural network, each phase change material in the cross-waveguide array can be initialized to the full-pass state of the optical signal, and calibration optical signals with the same intensity can be input into each input channel of the cross-waveguide array, then the optical power output from each output channel of the cross-waveguide array is collected, and the phase change materials in the cross-waveguide array are weight-calibrated according to this optical power and steps C and D described later. After the weight calibration of the phase change materials in the first cross-waveguide array is completed, each phase change material in the second cross-waveguide array can be initialized to the full-pass state of the optical signal, and calibration optical signals with the same intensity can be input into each input channel of the second cross-waveguide array, then the optical power output from each output channel of the second cross-waveguide array is collected, and the phase change materials in the second cross-waveguide array are weight-calibrated according to this optical power and steps C and D described later. After the weight calibration of the phase change materials in the second cross-waveguide array is completed, the above steps are repeated to weight-calibrate the phase change materials in the third cross-waveguide array until the weight calibration of the phase change materials in each cross-waveguide array in the optical neural network is completed.

[0133] In specific implementation, taking the weight calibration of the phase change materials in each cross-waveguide array in the optical neural network simultaneously as an example, the optical power output from each output channel corresponding to each cross-waveguide array can be collected, and the value of the minimum optical power among these optical powers is calibrated as a preset value. Among them, the preset value can be 1, for example.

[0134] Step C: Determine the target optical power of each phase change material according to the preset value, the minimum optical power, and the convolution kernel weight corresponding to each phase change material.

[0135] In specific implementation, according to the convolution kernel weights obtained after network training and the corresponding relationship between the preset value and the minimum optical power, the magnitude of the target optical power of each phase change material under a convolution kernel weight to be characterized can be deduced inversely.

[0136] Step D: Heat each phase change material respectively according to the target optical power of each phase change material so that its optical constant value matches the convolution kernel weight.

[0137] In specific implementation, each phase change material can be heated to the target optical power in sequence according to the target optical power of each phase change material, so that the optical constant values of each phase change material are consistent with the convolution kernels to be characterized, thereby realizing the weight calibration of each phase change material.

[0138] Exemplarily, the calibration process of the phase change material can be as follows: all phase change materials are in the state of no voltage applied (optical signal all-pass state). Calibration optical signals with the same intensity are input into each input port of the optical neural network in sequence, and the optical power of each output port of the optical neural network is recorded. The minimum optical power value is calibrated as 1. According to the convolution kernel weights of the model, calculate the corresponding optical power magnitude, and calibrate each phase change material in sequence to make its output optical power consistent with the expected magnitude. Since electricity is required for weight calibration of the phase change material, and once the calibration is completed, no electricity is required during subsequent use of the phase change material, near-zero power consumption inference can be achieved.

[0139] In another embodiment, the calibration at the software level is specifically reflected in the network training process. Specifically, the convolution kernel weights can be obtained through training by the following steps: During the iterative training process of the optical neural network, convolution kernel weights quantized with a set-size bit value are used in the forward propagation process, and floating-point numbers are used for gradient calculation to obtain convolution kernel weights in the reverse propagation process.

[0140] Here, the set-size bit value can be a set low-bit (bit) value. Since the accuracy of the front-end optical neural network is not particularly high (i.e., operations are performed at low precision), such as 4 bits or 8 bits, quantization-aware training is required. The set-size bit value can be, for example, 4 bits or 8 bits.

[0141] In this way, at the software level, the present application mainly adopts quantization-aware training to reduce the reduction in calculation accuracy caused by the transition of model weights from high precision to low precision. The quantization-aware training method integrates quantization operations into the training process. Low-bit quantized convolution kernel weights and activation values are used in the forward propagation, while floating-point numbers are still used for gradient calculation in the reverse propagation, so that the model gradually adapts to quantization errors during the training process and reduces accuracy loss.

[0142] To facilitate the understanding of the embodiments of the present application, the following will take the data to be processed as a to-be-processed picture as an example, and combine Figure 4 the optical neural network structure to illustrate the data processing flow of the present application: Traverse the image to be processed according to the data block size of 3×3. For the sub-data obtained from the current traversal, the initial digital domain signal corresponding to the sub-data can be encoded into a first optical domain signal, and the first optical domain signal includes sub-signals with 9 wavelengths (i.e., λ1~λ9). Input the 9 sub-signals into each row in the cross-waveguide array 1 respectively, and use the 9 memory-computation integrated units corresponding to each row to perform convolution processing on the 9 sub-signals respectively to obtain the convolution signals corresponding to each row. The convolution signals of each row form the convolution result of convolution layer 1. Use the photodetectors connected to each row in the cross-waveguide array 1 to perform photoelectric conversion to obtain the first current signal corresponding to each row. Input the first current signal of each row into the micro-ring modulator and MZM corresponding to the row in the non-linearity 1, and use an independent laser source to input a preset optical domain signal (i.e., the preset optical domain signals corresponding to wavelengths λ1~λ9 respectively in the non-linearity 1) to the MZM in each row in the non-linearity 1. Adjust the phase of the MZM and the refractive index of the micro-ring modulator through the first current signal, and perform non-linear modulation on the input preset optical domain signal based on the adjusted phase and refractive index to obtain non-linear results. That is, obtain the non-linear result 1 corresponding to the preset optical domain signal with wavelength λ1, the non-linear result 2 corresponding to the preset optical domain signal with wavelength λ2, ……, the non-linear result 9 corresponding to the preset optical domain signal with wavelength λ9. Input the 9 non-linear results into the only column in the cross-waveguide array 2 respectively, and use the 9 memory-computation integrated units corresponding to this column to perform convolution processing on the non-linear results respectively to obtain the target convolution signal. That is, input the non-linear results 1~9 into the 9 memory-computation integrated units corresponding to the only column in the cross-waveguide array 2 to perform convolution processing to obtain the target convolution signal. Use a photodetector connected to the cross-waveguide array 2 to perform photoelectric conversion on the target convolution signal to obtain a second current signal. Input the second current signal into the micro-ring modulator and MZM in the non-linearity 2, and use an independent laser source to input a preset optical domain signal to the MZM in the non-linearity 2. Adjust the phase of the MZM and the refractive index of the micro-ring modulator through the second current signal, and perform non-linear modulation on the input preset optical domain signal based on the adjusted phase and refractive index to obtain a second optical domain signal corresponding to the sub-data. Convert the second optical domain signal corresponding to the sub-data into a target digital domain signal, that is, obtain Figure 4 the target digital domain signal of a pixel point in the processing result of the optical neural network in Figure 4 . Through continuous traversal of the image to be processed, and inputting the sub-data obtained from each traversal into the optical neural network for the above-introduced convolution processing and non-linear processing, the target digital domain signal of a pixel point corresponding to each sub-data in the image to be processed in the processing result of the optical neural network is obtained. The target digital domain signals of each sub-data form

[0143] Corresponding to the embodiments of the foregoing data processing method, the present application also provides embodiments of a data processing apparatus.

[0144] The embodiments of the data processing apparatus of the present application can be applied to edge-side terminals. The apparatus embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logically meaningful apparatus, it is formed by the processor of its host terminal reading the corresponding computer program instructions in the non-volatile memory into the memory for running. At the hardware level, as Figure 5 shown, it is a hardware structure diagram of the terminal where the data processing apparatus 600 of the present application is located. In addition to Figure 5 the processor, memory, network interface, and non-volatile memory shown, the terminal where the apparatus is located in the embodiments usually further includes other hardware according to the actual functions of the terminal, which will not be elaborated here.

[0145] Please refer to Figure 6 , which is a schematic diagram of a data processing apparatus provided by an embodiment of the present application, including: A traversal module 601, configured to traverse the data to be processed according to the rules of convolution calculation, and encode the initial digital domain signal corresponding to the current traversed sub-data into a first optical domain signal; A processing module 602, configured to input the first optical domain signal into an optical neural network for at least one convolution process and non-linear process to obtain a second optical domain signal of the sub-data on the output channel; the convolution kernel weights used in the convolution process are characterized by the optical constant values of the phase change materials in each memory-computation integrated unit set in the optical neural network; A classification module 603, configured to convert the second optical domain signals corresponding to the respective sub-data obtained by traversal into target digital domain signals and input them into a classification neural network to obtain a classification result of the data to be processed.

[0146] In a possible implementation manner, the optical neural network includes at least one hidden layer, and the hidden layer includes a cross-waveguide array for convolution processing and a microring modulation array for non-linear processing; the cross-waveguide array includes a plurality of memory-computation integrated units; When the processing module 602 inputs the first optical domain signal into the optical neural network for at least one convolution process and non-linear process to obtain the second optical domain signal of the sub-data on the output channel, it is configured to: Input the first optical domain signal into the first hidden layer in the optical neural network, and use each memory-computation integrated unit in the first hidden layer to perform a convolution process on the first optical domain signal to obtain a convolution result; Input the convolution result into the microring modulation array of the first hidden layer for non-linear processing to obtain a non-linear result; In the case where there is a next hidden layer, input the non-linear result into the next hidden layer to obtain the non-linear result output by the next hidden layer; Use the non-linear result output by the last hidden layer as the second optical domain signal of the sub-data on the output channel.

[0147] In a possible implementation manner, the memory-computation integrated unit at least includes input / output waveguides, directional couplers, and phase change materials; The multiple memory-computation integrated units are arranged according to a set structure. Each memory-computation integrated unit in a row or a column under the arrangement corresponds to a convolution kernel, and each convolution kernel weight of the convolution kernel is respectively characterized by the optical constant value of the phase change material of each memory-computation integrated unit; When the processing module 602 uses each memory-computation integrated unit in the first hidden layer to perform convolution processing on the first optical domain signal to obtain a convolution result, it is used for: For any convolution kernel of the first hidden layer, input the first optical domain signal into the memory-computation integrated units corresponding to the convolution kernel through the input waveguides corresponding to the convolution kernel; For any memory-computation integrated unit, use the directional coupler in the memory-computation integrated unit to couple the optical energy of the first optical domain signal at a set ratio, and perform multiplication processing using the convolution kernel weight characterized by the optical constant value of the phase change material to obtain the multiplication result of the memory-computation integrated unit and output it through the output waveguide; Determine the initial convolution result corresponding to the first optical domain signal under the convolution kernel according to the multiplication results output by the output waveguides of the memory-computation integrated units corresponding to the convolution kernel; Determine the convolution result of the first optical domain signal according to the initial convolution results corresponding to the first optical domain signal under each of the convolution kernels.

[0148] In a possible implementation manner, the microring modulation array includes a plurality of microring modulators. A doping region is provided in the microring modulator, and the doping region includes P-type semiconductor carriers and N-type semiconductor carriers; When the processing module 602 inputs the convolution result into the microring modulation array of the first hidden layer for non-linear processing to obtain a non-linear result, it is used for: Convert the convolution result into a current signal and input it into each microring modulator in the microring modulation array of the first hidden layer; Drive the change of the carrier concentration in the doped region by using the current signal, change the refractive index of the microring modulator, and perform nonlinear modulation based on the changed refractive index to obtain an initial modulation result; Obtain the nonlinear result based on the initial modulation results corresponding to the respective microring modulators.

[0149] In a possible implementation manner, the microring modulation array further includes a first preset modulator cascaded with each microring modulator; When performing nonlinear modulation based on the changed refractive index to obtain an initial modulation result, the processing module 602 is configured to: Input the current signal into the first preset modulator cascaded with each microring modulator to adjust the phase of the first preset modulator; Use an independent laser source set for each first preset modulator to input a preset optical domain signal into each first preset modulator respectively; Perform nonlinear modulation on the preset optical domain signal by using the adjusted phase and the changed refractive index to obtain the initial modulation result.

[0150] In a possible implementation manner, when encoding the initial digital domain signal corresponding to the sub-data into a first optical domain signal, the traversal module 601 is configured to: Convert the initial digital domain signal corresponding to the sub-data into an analog domain signal and input the analog domain signal into each second preset modulator; Perform multi-wavelength modulation on the analog domain signal by using each second preset modulator to obtain modulation signals corresponding to the respective second preset modulators; Use a wavelength division multiplexer associated with each second preset modulator to separate and replicate the modulation signal corresponding to each second preset modulator according to the signal wavelength to obtain a demultiplexed signal; Perform a dislocation process on the demultiplexed signal by using a delay line to obtain the first optical domain signal.

[0151] In a possible implementation manner, the apparatus further includes a first calibration module 604 for calibrating the weight characterized by the phase change material through the following steps: Initialize each phase change material in the optical neural network to an optical signal all-pass state and input calibration optical signals with the same intensity into each input channel of the optical neural network; Collect the optical powers output from each output channel corresponding to the cross-waveguide array in the optical neural network and calibrate the numerical value of the minimum optical power among the respective optical powers to a preset value; Determine the target optical power of each phase change material according to the preset value, the minimum optical power, and the convolution kernel weight corresponding to each phase change material; Heat each phase change material according to the target optical power of each phase change material, so that its optical constant value matches the convolution kernel weight.

[0152] In a possible implementation manner, the device further includes a second calibration module 605, which is used to train the convolution kernel weight in the following manner: During the iterative training of the optical neural network, the convolution kernel weight quantized by a set-size bit value is used in the forward propagation process, and the floating point number is used to calculate the gradient to obtain the convolution kernel weight in the backward propagation process.

[0153] The implementation processes of the functions and roles of each unit in the above device are specifically described in the implementation processes of the corresponding steps in the above method, and will not be elaborated here.

[0154] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present application. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0155] The embodiments of the subject matter and functional operations described in this specification can be implemented in the following: digital electronic circuits, tangible embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more modules in computer program instructions encoded on a tangible non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode and transmit information to a suitable receiver device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0156] The processes and logical flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform the corresponding functions by operating on input data and generating output. The processes and logical flows can also be performed by, for example, FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) of special logic circuits, and the apparatus can also be implemented as special logic circuits.

[0157] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, etc., or the computer will be operatively coupled to such mass storage devices to receive data therefrom or transfer data thereto, or both. However, a computer is not necessarily required to have such devices. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name just a few.

[0158] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as including semiconductor memory devices (such as EPROM, EEPROM, and flash memory devices), magnetic disks (such as internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special logic circuits.

[0159] Although this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of what is claimed, but rather are mainly used to describe the features of specific embodiments of a particular invention. Certain features described in multiple embodiments in this specification can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may operate in certain combinations as described above and are even initially claimed as such, one or more features from a claimed combination can in some cases be removed from that combination, and the claimed combination can be directed to a sub-combination or a variation of a sub-combination.

[0160] Similarly, although the operations are depicted in the drawings in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or sequentially, or that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Additionally, the separation of various system modules and components in the above-described embodiments should not be understood as required in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged into multiple software products.

[0161] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the acts recited in the claims can be performed in a different order and still achieve the desired result. Additionally, the processes depicted in the drawings are not necessarily in the particular order or sequential order shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0162] The foregoing is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the scope of protection of the present application.

Claims

1. A data processing method, characterized in that The method includes: Traversing the data to be processed according to the rules of convolution calculation. For the currently traversed sub-data, encoding the initial digital domain signal corresponding to the sub-data into a first optical domain signal; Inputting the first optical domain signal into an optical neural network for at least one convolution process and non-linear process to obtain a second optical domain signal of the sub-data on the output channel; the convolution kernel weights used in the convolution process are characterized by the optical constant values of the phase change materials in each memory-computation integrated unit set in the optical neural network; Converting the second optical domain signals corresponding to the respective sub-data obtained by traversal into target digital domain signals and inputting them into a classification neural network to obtain the classification result of the data to be processed.

2. The method according to claim 1, characterized in that, The optical neural network includes at least one hidden layer, and the hidden layer includes a cross-waveguide array for convolution processing and a microring modulation array for non-linear processing; the cross-waveguide array includes a plurality of memory-computation integrated units; The step of inputting the first optical domain signal into an optical neural network for at least one convolution process and non-linear process to obtain a second optical domain signal of the sub-data on the output channel includes: Inputting the first optical domain signal into the first hidden layer of the optical neural network, and using each memory-computation integrated unit in the first hidden layer to perform a convolution process on the first optical domain signal to obtain a convolution result; Inputting the convolution result into the microring modulation array of the first hidden layer for non-linear processing to obtain a non-linear result; In the case where there is a next hidden layer, inputting the non-linear result into the next hidden layer to obtain the non-linear result output by the next hidden layer; Taking the non-linear result output by the last hidden layer as the second optical domain signal of the sub-data on the output channel.

3. The method according to claim 2, wherein The memory-computation integrated unit at least includes input / output waveguides, a directional coupler, and a phase change material; The plurality of memory-computation integrated units are arranged according to a set structure. Each memory-computation integrated unit in a row or a column under the arrangement corresponds to a convolution kernel, and each convolution kernel weight of the convolution kernel is respectively characterized by the optical constant value of the phase change material of each memory-computation integrated unit; The step of using each memory-computation integrated unit in the first hidden layer to perform a convolution process on the first optical domain signal to obtain a convolution result includes: For any convolution kernel of the first hidden layer, inputting the first optical domain signal into each memory-computation integrated unit corresponding to the convolution kernel through the respective input waveguides corresponding to the convolution kernel; For any memory-computation integrated unit, using the directional coupler in the memory-computation integrated unit to couple the optical energy of the first optical domain signal at a set ratio, and performing a multiplication process using the convolution kernel weight characterized by the optical constant value of the phase change material to obtain the multiplication result of the memory-computation integrated unit and output it through the output waveguide; Determining the initial convolution result of the first optical domain signal under the convolution kernel according to the multiplication results output by the respective output waveguides of the memory-computation integrated units corresponding to the convolution kernel; Determine the convolution result of the first optical domain signal according to the initial convolution results corresponding to the first optical domain signal under each of the convolution kernels.

4. The method according to claim 2, characterized in that, The microring modulation array includes a plurality of microring modulators, a doping region is provided in the microring modulator, and the doping region includes P-type semiconductor carriers and N-type semiconductor carriers; The inputting the convolution result into the microring modulation array of the first hidden layer for non-linear processing to obtain a non-linear result includes: Convert the convolution result into a current signal and input it into each microring modulator in the microring modulation array of the first hidden layer; Use the current signal to drive the change in the carrier concentration in the doping region, change the refractive index of the microring modulator, and perform non-linear modulation based on the changed refractive index to obtain an initial modulation result; Based on the initial modulation results corresponding to each microring modulator, obtain the non-linear result.

5. The method according to claim 4, characterized in that, The microring modulation array further includes a first preset modulator cascaded with each microring modulator; The performing non-linear modulation based on the changed refractive index to obtain an initial modulation result includes: Input the current signal into the first preset modulator cascaded with each microring modulator to adjust the phase of the first preset modulator; Use an independent light source set for each first preset modulator to input a preset optical domain signal into each first preset modulator respectively; Perform non-linear modulation on the preset optical domain signal by using the adjusted phase and the changed refractive index to obtain the initial modulation result.

6. The method according to claim 1, characterized in that, The encoding the initial digital domain signal corresponding to the sub-data into a first optical domain signal includes: Convert the initial digital domain signal corresponding to the sub-data into an analog domain signal and input the analog domain signal into each second preset modulator; Use each of the second preset modulators to perform multi-wavelength modulation on the analog domain signal to obtain modulation signals corresponding to each of the second preset modulators; Use a wavelength division multiplexer associated with each second preset modulator to separate and replicate the modulation signal corresponding to each second preset modulator according to the signal wavelength to obtain a demultiplexed signal; Use a delay line to perform dislocation processing on the demultiplexed signal to obtain the first optical domain signal.

7. The method according to claim 1, wherein The method further includes the step of calibrating the weights characterized by the phase change material: Initialize each phase change material in the optical neural network to an optical signal all-pass state and input calibration optical signals with the same intensity into each input channel of the optical neural network; Collect the optical powers output from each output channel corresponding to the cross-waveguide array in the optical neural network, and calibrate the value of the minimum optical power among the optical powers to a preset value; Determine the target optical power of each phase change material according to the preset value, the minimum optical power, and the convolution kernel weight corresponding to each phase change material; Heat each phase change material according to the target optical power of each phase change material respectively so that its optical constant value matches the convolution kernel weight.

8. The method according to claim 1, wherein The convolution kernel weights are trained in the following manner: In the process of iteratively training the optical neural network, during the forward propagation process, the convolutional kernel weights quantized with a set - sized bit value are used, and during the back - propagation process, floating - point numbers are used to calculate the gradients to obtain the convolutional kernel weights.

9. A data processing device, characterized in that, The device includes: A traversal module, configured to traverse the data to be processed according to the rules of convolutional calculation. For the currently traversed sub - data, encode the initial digital - domain signal corresponding to the sub - data into a first optical - domain signal; A processing module, configured to input the first optical - domain signal into the optical neural network for at least one convolutional processing and non - linear processing to obtain a second optical - domain signal of the sub - data on the output channel; the convolutional kernel weights used in the convolutional processing are characterized by the optical constant values of the phase - change materials in each memory - in - processing unit set in the optical neural network; A classification module, configured to convert the second optical - domain signals corresponding to the respective sub - data obtained by traversal into target digital - domain signals and input them into the classification neural network to obtain the classification result of the data to be processed.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 8.

Images