A photonic convolutional neural network system

Abstract

The present invention discloses a photonic convolutional neural network system, which belongs to the field of optical neural network technology. The system includes dynamic convolution kernel modules corresponding one-to-one to each convolution layer in a pre-trained convolutional neural network model. The dynamic convolution kernel module includes a photonic synaptic device based on phase change material, and the convolution kernel weights of the corresponding convolution layer are non-volatilely stored in the photonic synaptic device. Before the dynamic convolution kernel module performs a convolution operation, the weights of the corresponding photonic synaptic devices therein are dynamically adjusted through volatile modulation according to the dynamic coefficients generated by a coefficient generation model according to the content of the input data, so as to flexibly adapt to changes in the input data. The present invention does not only rely on the non-volatile phase state of the phase change material for encoding, but further introduces volatile modulation coding on this basis, thereby greatly enriching the encoding expression capability of the dynamic convolution kernel module. Based on this, the feature extraction capability of the convolution kernel in the photonic convolutional neural network system provided by the present invention is relatively strong.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical neural networks, and more specifically, relates to a photonic convolutional neural network system. Background Art

[0002] Convolutional Neural Networks (CNNs) have achieved remarkable success in fields such as image classification and natural language processing. However, in real-world data processing, convolution operations, as prerequisites for CNNs, consume the majority of CNN computing power. With the explosive growth of CNN data volumes, the von Neumann bottleneck of traditional electronic chips has become increasingly prominent, making it increasingly difficult to meet the rapidly growing hardware system requirements of CNNs. Optical neuromorphic computing systems, leveraging the advantages of photons—high speed, parallelism, low crosstalk, low energy consumption, and high interconnect bandwidth—as well as the rapid development of integrated optoelectronics in recent years, have become a disruptive high-performance CNN computing architecture.

[0003] At present, traditional photonic computing chips mainly use silicon-based materials as the substrate to construct optical components, and realize active adjustment of devices through carrier dispersion effect or thermo-optical effect. However, the refractive index adjustment range of carrier dispersion effect is limited (~10 -3 ) and the phase modulation effect is not ideal; the response time of the thermo-optical effect is slow, usually on the order of milliseconds. In recent years, phase-change materials such as GST have attracted widespread attention in the field of integrated photonics due to their excellent optical programmable properties. Phase-change materials are integrated with silicon-based devices to construct photonic convolution accelerators, and the non-volatile nature of the phase-change material is used to store different convolution kernel weight values, realizing "in-memory computing" of convolution. Based on this, existing photonic convolutional neural network systems encode pre-trained convolution kernel weight mappings into the non-volatile phase state of the phase-change material. When data is input, the input data is convolved with the fixedly stored convolution kernel. However, in this method, the convolution kernel is fixed and cannot be adaptively adjusted according to the characteristics of the input data. The ability to extract different input data features is limited, which restricts performance in complex tasks. At the same time, the limited number of non-volatile states of the phase-change material restricts the value of the convolution kernel weight, which in turn restricts the expression of the convolution kernel and the feature extraction capability. Summary of the Invention

[0004] In response to the above defects or improvement needs of the existing technology, the present invention provides a photonic convolutional neural network system to solve the technical problem of limited convolution kernel feature extraction capability in the existing photonic convolutional neural network system.

[0005] In order to achieve the above object, the present invention provides a photonic convolutional neural network system, comprising: dynamic convolution kernel modules corresponding one-to-one to each convolution layer in a pre-trained convolutional neural network model;

[0006] The dynamic convolution kernel module includes: a photonic synaptic device array; the array stores pre-trained convolution kernel weights in the corresponding convolution layer; wherein the photonic synaptic devices in the array are photonic synaptic devices based on phase change materials; the convolution kernel weights are encoded into the corresponding non-volatile phase states of the photonic synaptic devices through non-volatile modulation, thereby achieving storage; one photonic synaptic device stores one convolution kernel weight; and the light transmittance of the photonic synaptic devices in different phase states is different.

[0007] The dynamic convolution kernel module is used to dynamically adjust the convolution kernel weights stored in the array before performing the convolution operation: receiving the dynamic coefficients corresponding to the convolution kernel weights stored in the array, and applying a pulse signal corresponding to the corresponding dynamic coefficient to each photonic synapse device for volatile modulation;

[0008] The dynamic convolution kernel module is also used to receive continuous signal light carrying data information to be convolved during the convolution operation. The continuous signal light is attenuated by the array and the attenuated signal light is output. The intensity of the output signal light is the result of the current convolution operation.

[0009] Among them, the dynamic coefficient is generated by inputting the data to be convolution operation into the corresponding pre-trained coefficient generation model; one dynamic convolution kernel module corresponds to one coefficient generation model, and the coefficient generation model is a deep learning model.

[0010] Further preferably, the coefficient generation model includes: a plurality of cascaded fully connected layers, an activation layer arranged between two adjacent fully connected layers, and a pooling layer connected before the first fully connected layer.

[0011] Further preferably, the training method of the coefficient generation model corresponding to each dynamic convolution kernel module includes:

[0012] The above-mentioned coefficient generation model is introduced in the training process of the convolutional neural network model. The coefficient generation model corresponding to each dynamic convolution kernel module corresponds one-to-one to each convolution layer in the convolutional neural network model. In the forward propagation process, before performing the convolution operation, each convolution layer generates the dynamic coefficients corresponding to the weights of each convolution kernel in the convolution layer based on the data input to the convolution layer through the corresponding coefficient generation model, and dynamically adjusts the corresponding convolution kernel weights in the convolution layer based on the obtained dynamic coefficients. In the backward propagation process, the parameters in the convolutional neural network model and each coefficient generation model are adjusted at the same time.

[0013] Further preferably, each weight in the same convolution kernel is stored in the same column of the array;

[0014] The dynamic convolution kernel module is further configured to dynamically adjust the size of each convolution kernel stored in the array before performing a convolution operation and after dynamically adjusting the convolution kernel weights stored in the array: selecting a number of photonic synaptic devices from each column of the array, maintaining the phase state of the selected photonic synaptic devices unchanged, and modulating the phase state of the unselected photonic synaptic devices to a fully crystalline state through a non-volatile modulation method, so as to adjust the size of the convolution kernel stored in the column to a corresponding optimal size;

[0015] Among them, a dynamic convolution kernel module also corresponds to a size selection model, which is a deep learning model;

[0016] The optimal size of each convolution kernel stored in the array is generated by inputting the data to be convolved into the corresponding size selection module; the size selection model is used to extract the features of the data to be convolved, and then map them to the optimal size of each convolution kernel stored in the array in the corresponding dynamic convolution kernel module.

[0017] Further preferably, the size selection model includes: a cascaded feature extraction module and a classifier; wherein the feature extraction module includes: one or more of an edge feature extraction unit, a texture feature extraction unit and a smoothness feature extraction unit.

[0018] Further preferably, the above-mentioned classifier is: a fully connected conditional network; wherein the fully connected conditional network includes: multiple cascaded fully connected layers, an activation layer arranged between two adjacent fully connected layers, and a softmax layer connected after the last fully connected layer.

[0019] Further preferably, the training method of the coefficient generation model and the size selection model corresponding to each dynamic convolution kernel module includes:

[0020] The above-mentioned coefficient generation model and size selection model are introduced in the training process of the convolutional neural network model. The coefficient generation model corresponding to each dynamic convolution kernel module corresponds one-to-one to each convolution layer in the convolutional neural network model; the size selection model corresponding to each dynamic convolution kernel module corresponds one-to-one to each convolution layer in the convolutional neural network model;

[0021] During the forward propagation process, before performing the convolution operation, each convolution layer generates the dynamic coefficients corresponding to the weights of each convolution kernel in the convolution layer based on the input data of the convolution layer through the corresponding coefficient generation model, and dynamically adjusts the corresponding convolution kernel weights in the convolution layer based on the obtained dynamic coefficients; and generates the optimal size of each convolution kernel in the convolution layer based on the input data of the convolution layer through the corresponding size selection model, so as to dynamically adjust the size of each convolution kernel in the convolution layer;

[0022] During the back-propagation process, the parameters in the convolutional neural network model, the coefficient generation models, and the size selection models are adjusted simultaneously.

[0023] Further preferably, the above-mentioned photonic convolutional neural network system also includes: a light source module, which is used to use an optical signal modulator to load the data to be currently convolved onto the optical signal, and obtain a continuous signal light carrying the data information to be currently convolved.

[0024] Further preferably, the above-mentioned photonic convolutional neural network system further includes: a photoelectric conversion module, a nonlinear activation module and a fully connected layer module;

[0025] The photoelectric conversion module is used to convert the signal light output by the dynamic convolution kernel module into an electrical signal and output it to the nonlinear activation module;

[0026] The nonlinear activation module is used to perform nonlinear activation processing on the electrical signal input by the photoelectric conversion module;

[0027] The fully connected layer module is used to calculate the nonlinear activation processing result output by the nonlinear activation module for the last time to obtain the final output result.

[0028] Further preferably, the photonic synapse device comprises: a substrate, a waveguide layer, a phase change material layer, a heating layer, a covering layer, and an electrode layer acting on the heating layer, which are arranged in sequence from bottom to top.

[0029] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0030] 1. The present invention provides a photonic convolutional neural network system, comprising dynamic convolution kernel modules corresponding one-to-one to each convolution layer in a pre-trained convolutional neural network model; wherein the dynamic convolution kernel module comprises a photonic synaptic device based on phase change material, in which the convolution kernel weights of the corresponding convolution layer are non-volatilely stored; before the dynamic convolution kernel module performs a convolution operation, the weights of the corresponding photonic synaptic devices therein are dynamically adjusted by volatile modulation according to the dynamic coefficients generated by a coefficient generation model according to the content of the input data, so as to flexibly adapt to changes in the input data; at the same time, the present invention does not only rely on the non-volatile phase state of the phase change material for encoding, but further introduces volatile modulation coding on this basis, effectively alleviating the problem that the existing photonic convolutional neural network system is severely limited by the number of non-volatile phase states of the phase change material, and greatly enriching the coding expression capability of the convolution kernel in the dynamic convolution kernel module; based on this, the feature extraction capability of the convolution kernel in the photonic convolutional neural network system provided by the present invention is relatively strong.

[0031] 2. Furthermore, in the photonic convolutional neural network system provided by the present invention, the coefficient generation model includes: first, using a pooling layer to extract local and global features, reducing the computational complexity while retaining important features; then using multiple cascaded fully connected layers to further gradually extract, combine and abstract these features, and introducing an activation layer between two adjacent fully connected layers, which enables the coefficient generation model to capture complex patterns in the input data, so that the coefficient generation model can adapt to inputs of different sizes, enhance the generalization ability of the model, generate coefficients suitable for adjusting the convolution kernel, and further improve the feature extraction ability of the photonic convolutional neural network system.

[0032] 3. Furthermore, the coefficient generation model adopted by the photonic convolutional neural network system provided by the present invention is obtained through collaborative training with the convolutional neural network model corresponding to the photonic convolutional neural network system. It can fully match the characteristics of the convolution kernel weights in the convolutional neural network model while paying attention to the data to be convolutionally operated, thereby further improving the feature extraction capability of the photonic convolutional neural network system.

[0033] 4. Furthermore, in the photonic convolutional neural network system provided by the present invention, before the dynamic convolution kernel module performs a convolution operation, it also dynamically adjusts the convolution kernel size stored in the photonic synapse device array through a non-volatile modulation method according to the optimal size of the convolution kernel obtained by the size selection model according to the characteristics of the input data, so that the dynamic convolution kernel module can extract data information at different optimal levels, further improving the ability of the photonic convolutional neural network system to recognize and express multi-scale features.

[0034] 5. Furthermore, in the photonic convolutional neural network system provided by the present invention, the feature extraction module of the size selection model includes: one or more of an edge feature extraction unit, a texture feature extraction unit, and a smoothness feature extraction unit. By extracting the statistical characteristics of the complexity and detail level of the input data, the frequency of use of convolution kernels of different sizes can be dynamically adjusted according to the input data of different complexities and features, and the optimal size of the convolution kernel can be determined more accurately, thereby further improving the ability of the photonic convolutional neural network system to recognize and express multi-scale features.

[0035] 6. Furthermore, the size selection model adopted by the photonic convolutional neural network system provided by the present invention is obtained through collaborative training with the coefficient generation model and the convolutional neural network model corresponding to the photonic convolutional neural network system. It can fully match the characteristics of the convolution kernel weights in the convolutional neural network model while paying attention to the data to be convolutionally operated, thereby further improving the feature extraction capability of the photonic convolutional neural network system.

[0036] 7. Furthermore, in the photonic convolutional neural network system provided by the present invention, the photonic synaptic device preferably adopts a photonic synaptic device based on electrical regulation, which has a fast regulation speed and can be regulated at the nanosecond or microsecond level; has strong resistance to optical noise interference and good stability; has high regulation accuracy, and can accurately control and adjust optical parameters by electric heating elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the structure of a photonic convolutional neural network system provided in an embodiment of the present invention.

[0038] Figure 2 A schematic diagram of the structure of a dynamic convolution module provided in an embodiment of the present invention.

[0039] Figure 3 A schematic structural diagram of a phase-change photonic synaptic device based on electrical modulation provided by an embodiment of the present invention.

[0040] Figure 4 A diagram showing the non-volatile modulation performance of the phase-change photonic synapse device provided in an embodiment of the present invention.

[0041] Figure 5 A diagram showing the volatility modulation performance of the phase-change photonic synapse device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0043] In order to achieve the above object, the present invention provides a photonic convolutional neural network system, comprising: dynamic convolution kernel modules corresponding one-to-one to each convolution layer in a pre-trained convolutional neural network model;

[0044] The dynamic convolution kernel module includes: a photonic synaptic device array; the photonic synaptic devices in the array are photonic synaptic devices based on phase change materials; the array stores pre-trained convolution kernel weights in the corresponding convolution layer; wherein the convolution kernel weights are encoded into the corresponding non-volatile phase state of the photonic synaptic device through non-volatile modulation, thereby achieving storage; each photonic synaptic device stores one convolution kernel weight; and the light transmittance of the photonic synaptic device in different phase states is different.

[0045] The dynamic convolution kernel module is used to dynamically adjust the convolution kernel weights stored in the array before performing the convolution operation: receiving the dynamic coefficients corresponding to the convolution kernel weights stored in the array, and applying a pulse signal corresponding to the corresponding dynamic coefficient to each photonic synapse device for volatile modulation;

[0046] The dynamic convolution kernel module is also used to receive continuous signal light carrying data information to be convolved during the convolution operation. The continuous signal light is attenuated by the array and the attenuated signal light is output. The intensity of the output signal light is the result of the current convolution operation.

[0047] Among them, the dynamic coefficient is generated by inputting the data to be convolution operation into the corresponding pre-trained coefficient generation model; one dynamic convolution kernel module corresponds to one coefficient generation model, and the coefficient generation model is a deep learning model.

[0048] It should be noted that the data to be convolutionally operated depends on the specific tasks of the photonic convolutional neural network system (such as natural language processing tasks, image classification tasks, etc.), and can be voice data, image data, etc.

[0049] It should be noted that the input layer dimension of the coefficient generation model is the data dimension of the corresponding dynamic convolution kernel module to be convolution operation, and the output layer dimension is the number of convolution kernel weights stored in the array in the corresponding dynamic convolution kernel module or the number of convolution kernels stored in the array in the corresponding dynamic convolution kernel module. When the dimension of the coefficient generation model is the number of convolution kernel weights stored in the array in the corresponding dynamic convolution kernel module, the generation of the coefficient generation model is directly the dynamic coefficient corresponding to the convolution kernel weight stored in the array. When the dimension of the coefficient generation model is the number of convolution kernels stored in the array in the corresponding dynamic convolution kernel module, the generation of the coefficient generation model is directly the dynamic coefficient corresponding to each convolution kernel stored in the array; at this time, the coefficients of the convolution kernel weights belonging to the same convolution kernel are all dynamic coefficients corresponding to the convolution kernel, thereby obtaining the dynamic coefficients corresponding to each convolution kernel stored in the array.

[0050] Through the above different settings, the dynamic convolution kernel module can perform volatile modulation on each convolution kernel weight stored in its array, and can also perform volatile modulation on each convolution kernel stored in its array.

[0051] The dynamic convolution kernel module uses dynamic coefficients generated according to the input data to be convolutionally operated to act on the convolution kernel weights pre-stored in the array. After volatile modulation of the convolution kernel weights, a new dynamic convolution kernel is formed. The dynamically generated convolution kernel has stronger feature detection capabilities, thereby improving the feature extraction capabilities of the convolution network.

[0052] It should be noted that the photonic synapse devices (also known as optical synapse devices) in the aforementioned photonic synapse device array are phase-change material-based photonic synapse devices. Specifically, they can be electrically modulated or optically modulated, without limitation. For example, an electrically modulated phase-change photonic synapse device comprises, from bottom to top, a substrate, a waveguide layer, a phase-change material layer, a heating layer, a covering layer, and an electrode layer acting on the heating layer. Preferably, the heating layer of the phase-change optical synapse device completely covers the phase-change material layer to ensure a significant volatile modulation effect while not affecting the long-term maintenance of the non-volatile state. The optically modulated phase-change photonic synapse device differs from the electrically modulated phase-change photonic synapse device only in that it does not include the electrode layer acting on the heating layer; otherwise, the structures are identical. The optically modulated phase-change photonic synapse device uses optical modulation by inputting modulated light through the waveguide to modulate the phase-change material. The modulated light power is higher than the optical signal power.

[0053] When the photonic synapse devices in the photonic synapse device array are electrically modulated phase-change photonic synapse devices, the pulse signal used by the dynamic convolution kernel module to perform volatility modulation is an electrical pulse signal. When the photonic synapse devices in the photonic synapse device array are optically modulated phase-change photonic synapse devices, the pulse signal used by the dynamic convolution kernel module to perform volatility modulation is an optical pulse signal.

[0054] For the pulse signal applied by the dynamic convolution kernel module during volatile modulation, its parameters (such as amplitude, pulse width, etc.) correspond to the corresponding dynamic coefficient. The two can have a linear correspondence, an exponential correspondence, a logarithmic correspondence, or any combination of the above correspondences, etc., which are not limited here.

[0055] It should be noted that the data to be convolved is divided into multiple data segments using a sliding window method, with the window size consistent with the convolution kernel size in the dynamic convolution kernel module. Each data segment is loaded onto an optical signal and input into the array of the dynamic convolution kernel module. For example, if the weights for the same convolution kernel are stored in the same column of the array, and the convolution kernel size is m, then m optical signals are used, each loaded with a value from the data segment. During the convolution operation, the m optical signals are input one-to-one into the rows of the array storing the convolution kernel weights, completing the convolution operation between the current data segment and each convolution kernel in the dynamic convolution kernel module (the output of each column is the result of the convolution operation between the data segment and the corresponding convolution kernel). The case where the weights for the same convolution kernel are stored in the same row of the array is similar to the case where the weights for the same convolution kernel are stored in the same column of the array, except that the m optical signals are input one-to-one into the columns of the array storing the convolution kernel weights, and the output of each row is the result of the convolution operation between the data segment and the corresponding convolution kernel.

[0056] It should be noted that there are many coefficient generation models that can be used, such as CNN, Transformer, etc., which are not limited here. Preferably, in an optional embodiment, the coefficient generation model includes: multiple cascaded fully connected layers, an activation layer arranged between two adjacent fully connected layers, and a pooling layer connected before the first fully connected layer (which can be a maximum pooling layer, an average pooling layer, a global maximum pooling layer, a global average pooling layer, etc., which are not limited here). Specifically, for the input data to be convolutional, the pooling layer, as a feature extraction layer, will reduce the input feature space dimension and extract features, and then further map the features to the dynamic coefficients corresponding to the weights of each convolution kernel stored in the array after passing through the fully connected layer and the nonlinear layer.

[0057] It should be noted that there are multiple training methods for the above-mentioned coefficient generation model, which can be trained separately or in conjunction with the convolutional neural network model. In order to further improve the feature extraction capability of the photonic convolutional neural network system, it is preferred to adopt a method of collaborative training with the convolutional neural network model. Preferably, in an optional embodiment, the training method of the coefficient generation model corresponding to each dynamic convolution kernel module includes:

[0058] The above-mentioned coefficient generation model is introduced in the training process of the convolutional neural network model. The coefficient generation model corresponding to each dynamic convolution kernel module corresponds one-to-one to each convolution layer in the convolutional neural network model. In the forward propagation process, before performing the convolution operation, each convolution layer generates the dynamic coefficients corresponding to the weights of each convolution kernel in the convolution layer based on the data input to the convolution layer through the corresponding coefficient generation model, and dynamically adjusts the corresponding convolution kernel weights in the convolution layer based on the obtained dynamic coefficients. In the backward propagation process, the parameters in the convolutional neural model and each coefficient generation model are adjusted at the same time.

[0059] Preferably, when training the coefficient generation module, approximately uniform initial values are used for the parameters in the coefficient generation module at the beginning of training, so that more convolution kernels can be optimized simultaneously at the beginning of training to promote the learning of all convolution kernels.

[0060] In an optional embodiment, the weights in the same convolution kernel are stored in the same column of the array;

[0061] The dynamic convolution kernel module is further configured to dynamically adjust the size of each convolution kernel stored in the array before performing a convolution operation and after dynamically adjusting the convolution kernel weights stored in the array: selecting a number of photonic synaptic devices from each column of the array, maintaining the phase state of the selected photonic synaptic devices unchanged, and modulating the phase state of the unselected photonic synaptic devices to a fully crystalline state through a non-volatile modulation method, so as to adjust the size of the convolution kernel stored in the column to a corresponding optimal size;

[0062] Among them, a dynamic convolution kernel module also corresponds to a size selection model. The size selection model is a deep learning model. Its input layer dimension is the data dimension to be convolution operation of the corresponding dynamic convolution kernel module, and the output layer dimension is the number of convolution kernels stored in the array of the corresponding dynamic convolution kernel module.

[0063] The optimal size of each convolution kernel stored in the array is generated by inputting the data to be convolved into the corresponding size selection module; the size selection model is used to extract the features of the data to be convolved (such as one or more of edge features, texture features and smoothness features), and then map them to the optimal size of each convolution kernel stored in the array in the corresponding dynamic convolution kernel module.

[0064] It should be noted that there are many size selection models that can be used, such as CNN, Transformer, etc., which are not limited here. Preferably, in an optional embodiment, the size selection model includes: a cascaded feature extraction module and a classifier; wherein the feature extraction module includes: one or more of an edge feature extraction unit, a texture feature extraction unit, and a smoothness feature extraction unit. The statistical features of the complexity and detail level of the input data are extracted by one or more of the edge feature extraction unit, the texture feature extraction unit, and the smoothness feature extraction unit. wherein, the edge feature extraction unit can use an edge detection operator to extract edge features. In an optional embodiment, the edge detection operator can select a Roberts operator based on the first-order derivative, a Prewitt operator based on the first-order derivative, a Sobel operator based on the first-order derivative, a Laplacian operator based on the second-order derivative, a Canny operator based on non-differentiation, etc., to detect discontinuous edge information in the input data. The texture feature extraction unit may use a texture detection operator to extract texture features. In an optional embodiment, the texture detection operator may select a gray-level co-occurrence matrix, a local binary pattern, a Gabor filter, a Fourier transform, a wavelet transform, or the like to detect texture information present in the input data. The smoothness feature extraction unit may use methods such as local variance or global variance to perform calculations to detect local or global complexity information of the input data, thereby extracting smoothness features of the input data.

[0065] The scale selection model can dynamically adjust the usage frequency of convolution kernels of different sizes according to the input data of different complexity and features.

[0066] It should be noted that a variety of classifiers can be used, such as fully connected neural networks, support vector machines, decision trees, etc., which are not limited here. In one optional embodiment, the above classifier is: a fully connected conditional network; wherein the fully connected conditional network includes: multiple cascaded fully connected layers, an activation layer arranged between two adjacent fully connected layers, and a softmax layer connected after the last fully connected layer; wherein the activation layer can be a ReLU activation layer, a Sigmoid activation layer, a tanh activation layer, etc., which are not limited here.

[0067] It should be noted that the data to be convolved is divided into multiple data segments using a sliding window method. The window size is consistent with the convolution kernel size before dynamic rescaling in the dynamic convolution kernel module. Each time, a data segment is loaded onto the optical signal and input into the array of the dynamic convolution kernel module. Taking the example of each weight in the same convolution kernel stored in the same column of the array, let the convolution kernel size before dynamic rescaling be m, then m optical signals are used, each loaded with a value from the data segment. When performing the convolution operation, the m optical signals are input one-to-one into each row of the array storing the convolution kernel weights, completing the convolution operation between the current data segment and each convolution kernel in the dynamic convolution kernel module (the output of each column is the convolution result of the data segment and the corresponding convolution kernel). The situation where the weights in the same convolution kernel are stored in the same row of the array is similar to the situation where the weights in the same convolution kernel are stored in the same column of the array. The difference is that the m optical signals are input one by one into the columns of the array that store the convolution kernel weights, and the output of each row is the convolution operation result of the data segment and the corresponding convolution kernel.

[0068] It should be noted that there are multiple training methods for the size selection model. It can be trained independently or in conjunction with the convolutional neural network model. In order to further improve the feature extraction capability of the photonic convolutional neural network system, it is preferred to adopt a method of co-training with the convolutional neural network model. In one optional embodiment, the training methods for the coefficient generation model and the size selection model corresponding to each dynamic convolution kernel module include:

[0069] The above-mentioned coefficient generation model and size selection model are introduced in the training process of the convolutional neural network model. The coefficient generation model corresponding to each dynamic convolution kernel module corresponds one-to-one to each convolution layer in the convolutional neural network model; the size selection model corresponding to each dynamic convolution kernel module corresponds one-to-one to each convolution layer in the convolutional neural network model;

[0070] During the forward propagation process, before performing the convolution operation, each convolution layer generates the dynamic coefficients corresponding to the weights of each convolution kernel in the convolution layer based on the input data of the convolution layer through the corresponding coefficient generation model, and dynamically adjusts the corresponding convolution kernel weights in the convolution layer based on the obtained dynamic coefficients; and generates the optimal size of each convolution kernel in the convolution layer based on the input data of the convolution layer through the corresponding size selection model, so as to dynamically adjust the size of each convolution kernel in the convolution layer;

[0071] During the back-propagation process, the parameters in the convolutional neural network model, the coefficient generation models, and the size selection models are adjusted simultaneously.

[0072] In an optional embodiment, the above-mentioned photonic convolutional neural network system also includes: a light source module, which is used to use an optical signal modulator to load the data to be currently convolved onto the optical signal, and obtain a continuous signal light carrying the data information to be currently convolved.

[0073] It should be noted that optical signal modulators can precisely control optical signals based on different physical mechanisms (such as electro-optic effect, acousto-optic effect, etc.). Their types include but are not limited to: visible light modulators, electro-optic Mach-Zehnder modulators, microring resonators, acousto-optic modulators, etc., and the present invention does not limit this.

[0074] In an optional embodiment, the photonic convolutional neural network system further includes: a photoelectric conversion module, a nonlinear activation module and a fully connected layer module;

[0075] The photoelectric conversion module is used to convert the signal light output by the dynamic convolution kernel module into an electrical signal and output it to the nonlinear activation module;

[0076] The nonlinear activation module is used to perform nonlinear activation processing on the electrical signal input by the photoelectric conversion module;

[0077] The fully connected layer module is used to calculate the nonlinear activation processing result output by the nonlinear activation module for the last time to obtain the final output result.

[0078] It should be noted that the operating principle of the photoelectric conversion module is based on the photoelectric effect. When light strikes a semiconductor material, photons strike electrons in the semiconductor, causing them to transition to the conduction band, forming hole-electron pairs. These carrier pairs then form a directional current under an applied bias. The types of photodetectors used in the photoelectric conversion module include, but are not limited to, photodiodes, phototransistors, and avalanche photodiodes, and are not limited to these types in this disclosure.

[0079] The nonlinear activation module and the fully connected layer module can be existing hardware modules or software modules implemented on a host computer, without limitation. The nonlinear activation module can be a ReLU activation module, a Sigmoid activation module, a Tanh activation module, etc., without limitation.

[0080] In summary, the present invention provides a photon dynamic convolutional neural network system based on optical phase change materials. By combining non-volatile modulation methods and volatile modulation methods, and constructing coefficient generation models, size selection models and dynamic convolution modules, it can not only give full play to the non-volatile advantages of phase change materials in constructing photon convolution kernels, but also effectively improve the flexibility of photon convolution kernels. Photon convolution kernels can be adaptively and dynamically generated in a nonlinear manner according to the content of input data, overcoming the disadvantage that traditional static convolution kernels cannot adapt to changing inputs, and realizing flexible input adaptive dynamic convolution processing. In addition, when performing convolution operations, since the photon convolution kernel does not rely solely on the non-volatile phase state of the phase change material for encoding, but introduces volatile modulation coding on this basis, it greatly enriches the encoding expression capability of the photon convolution kernel, and thus can effectively alleviate the problem that the traditional photon convolution neural network system is severely limited by the number of non-volatile phase states of the phase change material. The dynamic adjustment of the convolution kernel size also greatly improves the ability of the photon convolution neural network system to recognize and express multi-scale features, retaining data information at different levels to improve network robustness. The photonic convolutional neural network system proposed in this application is also very computationally efficient. It can increase the network complexity without increasing the network depth or width, which is beneficial to improving the overall performance of the photonic convolutional neural network system, especially the classification accuracy of image data sets in image classification tasks. At the same time, it helps to promote the application of photonic convolutional neural network systems in larger-scale and more complex task scenarios.

[0081] In order to further illustrate the photonic convolutional neural network system provided by the present invention, a specific embodiment is described below in detail:

[0082] like Figure 1 As shown, this embodiment provides a photonic convolutional neural network system for realizing the functions of a convolutional neural network model; the convolutional neural network model includes: multiple cascaded convolutional layers, a nonlinear activation layer arranged between two adjacent convolutional layers, and a fully connected layer connected after the last convolutional layer.

[0083] The photonic convolutional neural network system provided in this embodiment includes: a computer module, a light source module, a dynamic convolution module and a photoelectric conversion module;

[0084] The computer module includes a training module, an input control module, P coefficient generation models, P size selection models, a coefficient cache module, a size cache module, a nonlinear activation module, and a fully connected layer module. P is the number of convolutional layers in the convolutional neural network model. Each convolutional layer in the convolutional neural network model corresponds to a coefficient generation model and a size selection model. The coefficient generation model corresponding to the i-th convolutional layer is denoted as the i-th coefficient generation model; the size selection model corresponding to the i-th convolutional layer is denoted as the i-th size selection model; i = 1, 2, 3, ..., P.

[0085] The training module is used to perform end-to-end training on the convolutional neural network model, each coefficient generation model, and each size selection model using a training set that performs corresponding tasks. Specifically, the training includes: in the forward propagation process, before each convolution layer performs a convolution operation, the corresponding coefficient generation model generates dynamic coefficients corresponding to the weights of each convolution kernel in the convolution layer based on the input data of the convolution layer, and dynamically adjusts the corresponding convolution kernel weights in the convolution layer based on the obtained dynamic coefficients; the corresponding size selection model generates the optimal size of each convolution kernel in the convolution layer based on the input data of the convolution layer, so as to dynamically adjust the size of each convolution kernel in the convolution layer; in the backward propagation process, the parameters in the convolutional neural network model, each coefficient generation model, and each size selection model are adjusted simultaneously. After training is completed, the convolution kernel weights in each convolution layer of the convolutional neural network model are stored, and the weight parameters of the last fully connected layer in the stored convolutional neural network model are loaded into the fully connected layer module.

[0086] There are also P dynamic convolution modules, each corresponding one-to-one to each convolutional layer in the convolutional neural network model. The dynamic convolution module corresponding to the i-th convolutional layer is denoted as the i-th dynamic convolution module. There is a one-to-one correspondence between the i-th dynamic convolution module, the i-th coefficient generation model, and the i-th size selection model. The dynamic convolution kernel module includes: a photonic synapse device array; the array stores the pre-trained convolution kernel weights for the corresponding convolution layer; the convolution kernel weights are encoded into the corresponding non-volatile phase state of the photonic synapse device through non-volatile modulation, thereby achieving storage; each photonic synapse device stores one convolution kernel weight; and the light transmittance of the photonic synapse device in different phase states varies.

[0087] In this embodiment, the data to be convolved is image data as an example. Figure 2 The following is a schematic diagram of the structure of the dynamic convolution module. For a dynamic convolution layer with weights [C1×C2×k×k], there are C1 layers with a size of [C2×k i ×k j] fixed convolution kernel (pre-trained convolution kernel). The coefficient generation model generates a set of dynamic coefficients based on the input features. According to the correspondence between the coefficients and the volatile modulation excitation, the weights are dynamically modulated with different levels or degrees of volatile modulation to enhance or suppress specific input-related features. The size selection model generates a set of size parameters based on the input information. It isolates some convolution kernel weights at a fully crystalline level encoded with high light attenuation to change the convolution kernel size and enhance the recognition of features at different scales. The correspondence between the dynamic coefficients and the volatile modulation excitation can be a combination of one or more of linear correspondence, exponential correspondence, and logarithmic correspondence.

[0088] During network inference, for each input image, the coefficient generation module generates a set of static convolution kernel coefficients. Using the excitation correspondence, the dynamic convolution module storing the static convolution kernel is regulated using volatile modulation. Volatile modulation is superimposed on non-volatile phase modulation to dynamically adjust the convolution kernel weights for that image. For each input image, the size selection module generates a set of static convolution kernel size parameters. Using the association, the static convolution kernel of the dynamic convolution module is modulated using non-volatile modulation. Phase-change optical devices not required for convolution are encoded as high attenuation levels, enabling dynamic adjustment of the convolution kernel size for that image. The processing time for each input image matches the time when the volatile modulation takes effect. After the dynamic convolution operation completes for each input image, the volatile modulation portion of the dynamic convolution module quickly deactivates, and the portion encoded as high attenuation is reset to its initial non-volatile value. This non-volatile value remains unchanged until the next input image is input.

[0089] In this embodiment, the dynamic convolution module has multiple photonic synaptic devices based on phase change materials, which correspond one to one with the convolution kernel weights of the corresponding convolution layer. Taking an electrically modulated phase change photonic synaptic device as an example, its device structure is as follows: Figure 3 As shown, the device specifically comprises, from bottom to top, a substrate, a waveguide layer, a phase-change material layer, a heating layer, a cover layer, and an electrode layer acting on the heating layer. The electrode layer is connected to an external power source and applies electrical pulses to the heating layer to regulate its temperature. As light propagates through the waveguide layer, it is controlled by both the waveguide layer and the phase-change material layer. The heating layer of the phase-change optical synapse device should be dimensioned to ensure a significant volatile modulation effect while maintaining the long-term non-volatile state.

[0090] The encoding methods for the photonic synaptic device in the dynamic convolution module include non-volatile modulation and volatile modulation. The non-volatile modulation method is used to encode the static convolution kernel (pre-trained convolution kernel) of the dynamic convolution module. The temperature control of the heating layer is used to modulate the phase change material layer. The static convolution kernel parameters trained by the computer module are stored in the non-volatile phase of the phase change material. The modulation of the transmittance by this phase state is the non-volatile modulation of the waveguide layer. The volatile modulation method is used to encode the adaptive input of the dynamic convolution module. Based on the static convolution kernel coefficients generated by the coefficient generation module through either thermo-optical effect or carrier dispersion effect, the waveguide layer is subjected to varying degrees of volatile modulation, thereby generating volatile modulation of the transmitted light in the waveguide layer.

[0091] The phase-change photonic synaptic device of the dynamic convolution module (a photonic synaptic device based on phase-change materials) has x programmable non-volatile phase levels that can transmit light, which are used to construct a static convolution kernel; each non-volatile phase has y volatile modulation levels for dynamically modulating the static convolution kernel; in addition, it also has a fully crystalline level with high light attenuation.

[0092] The optical transmittance of the fully crystalline state with high optical attenuation should be less than -60 dB. The optical transmittance of the x non-volatile phase states and the y volatile modulation levels within each non-volatile phase state should be greater than -40 dB. The optical transmittance calculation formula is: dB = 10log(P1 / P0), where P0 and P1 are the optical powers before and after passing through the phase-change optical synapse device, respectively. According to the above formula, when the optical transmittance is less than -60 dB, light transmission is considered completely blocked. When the optical transmittance differs by 20 dB, a significant difference in optical transmission is considered.

[0093] The transmittance of the x non-volatile phase levels and the y volatile modulation levels of each non-volatile phase of the phase change optical synaptic device should be normalized to [W min , W max ], where the minimum weight W min Corresponding to the lowest device transmittance, the maximum weight W max Corresponding to the highest device transmittance, the transmittance levels of the remaining devices correspond to different weights within this weight range, and are thus substituted into the computer module for training.

[0094] The selection result of the size selection module is associated with the fully crystalline level with high light attenuation, and the phase change optical synapse device corresponding to the convolution kernel position that does not need to play a convolution effect is encoded as the fully crystalline level with high light attenuation.

[0095] like Figure 4 and Figure 5The figures show the non-volatile modulation performance and volatile modulation performance of a phase-change photonic synaptic device. As can be seen from the figure, under different electrical modulation excitations, the phase-change photonic synaptic device can achieve non-volatile control and volatile control, and is used in the dynamic convolution module of the photonic convolutional neural network system. Figure 4 As shown, a voltage pulse is applied to the metal electrode, and the ratio of the crystalline phase to the amorphous phase of the phase change material layer is controlled by modulating the amplitude and / or pulse width of the voltage pulse to achieve non-volatile multi-state regulation of the phase change optical synapse device. It is worth noting that by regulating the amplitude and / or pulse width of the voltage pulse applied to the metal electrode, the applied energy can be precisely controlled to control the precise encoding of the non-volatile phase state of the phase change material. Figure 5 As shown, when the phase change material layer is in a certain non-volatile phase state, by applying a lower voltage pulse (below the energy threshold for inducing non-volatile phase change) to the metal electrode, the amplitude and / or pulse width of the modulated voltage pulse can be controlled to control the degree of volatility modulation of the waveguide layer, such as the thermo-optical effect, thereby achieving different levels of volatility modulation at a certain non-volatile phase state level.

[0096] The following is the complete workflow of the photonic convolutional neural network system:

[0097] In the inference phase, raw data is input into the convolutional neural network model for calculation. The specific process includes:

[0098] Perform the convolution operation corresponding to the first convolutional layer of the convolutional neural network model:

[0099] The input control module outputs the original data as the data to be currently subjected to convolution operation to the light source module, the first coefficient generation model and the first size selection module respectively;

[0100] After receiving the data to be convolutionally operated, the first coefficient generation model generates dynamic coefficients corresponding to the convolution kernel weights stored in the array within the first dynamic convolution module, and outputs the dynamic coefficients to the first dynamic convolution module through the coefficient cache module;

[0101] After receiving the data to be convolutionally operated, the first size selection module generates the optimal size of each convolution kernel stored in the array of the first dynamic convolution module, and outputs the optimal size to the first dynamic convolution module through the size cache module;

[0102] The first dynamic convolution module is configured to dynamically adjust the convolution kernel weights stored in the array before performing a convolution operation by receiving the dynamic coefficients corresponding to the convolution kernel weights stored in the array within the first dynamic convolution module input by the first coefficient generation module, and applying a pulse signal corresponding to the corresponding dynamic coefficient to each photonic synapse device for volatile modulation; dynamically adjust the size of each convolution kernel stored in the array by receiving the optimal size of each convolution kernel stored in the array within the first dynamic convolution module input by the first size selection module; select a number of photonic synapse devices from each column of the array, keep the phase state of the selected photonic synapse devices unchanged, and modulate the phase state of the unselected photonic synapse devices to a completely crystalline state through non-volatile modulation, so as to adjust the size of the convolution kernel stored in the column to the corresponding optimal size;

[0103] The light source module is used to, after receiving the data to be convolved, use the optical signal modulator to load the data to be convolved onto the optical signal, obtain a continuous signal light carrying the data information to be convolved, and output it to the first dynamic convolution module;

[0104] The first dynamic convolution module is further configured to receive, during a convolution operation, a continuous signal light input from the light source module carrying data information to be convolved, attenuate the continuous signal light under the action of the array, output the attenuated signal light, and output it to the photoelectric conversion module; the intensity of the output signal light is the result of the current convolution operation;

[0105] The photoelectric conversion module is used to convert the signal light carrying the convolution operation result information output by the first dynamic convolution module into an electrical signal, obtain the convolution operation result of the current layer, and output it to the nonlinear activation module;

[0106] The nonlinear activation module is used to perform nonlinear activation processing on the convolution operation result input by the photoelectric conversion module, and output the result after nonlinear processing to the input control module; wherein, the nonlinear activation module performs nonlinear activation processing on the electrical signal transmitted by the photoelectric conversion module through any of the following functions: Sigmoid function, tanh function, ReLU function, LeakyReLU function.

[0107] The input control module outputs the result of the nonlinear activation processing input by the nonlinear activation module as the data to be currently convolutionally operated to the light source module, the second coefficient generation module, and the second size selection module, respectively, to perform the convolution operation corresponding to the second convolution layer of the convolutional neural network model, and so on;

[0108] After completing the convolution operation of the P-layer convolutional layer, the nonlinear activation module outputs the result of its current nonlinear activation processing to the fully connected layer module to obtain the final calculation result.

[0109] It should be noted that the above-mentioned computer module can be a computer, server, workstation, etc. The computer module performs simulation through the simulation system loaded thereon, and uses the pre-collected data set to train the convolutional neural network model, coefficient generation module and size selection module. In some embodiments, the computer module can use the Python programming language to implement the above-mentioned functions, and use Python scripts or programs to control the modulation and weighting process of the entire photonic convolutional neural network system and the operating logic of each module. The Python programming language provides a rich interface and library for interacting with hardware devices. In addition to the Python programming language, other methods and tools can also be used to control the modulation and weighting of signals and the operating logic of each module, including but not limited to: MATLAB, Simulink, C / C++ programming, application-specific integrated circuits (ASICs), etc., which are not limited in the embodiments of this application.

[0110] In order to further illustrate the construction process of the photonic convolutional neural network system provided by the present invention, a specific embodiment is described in detail below:

[0111] In this embodiment, the steps of the method for constructing a photonic convolutional neural network system based on optical phase change materials mainly include steps S1 to S5, and each step is introduced in detail below.

[0112] Step S1: Using a vertical grating coupling test platform, obtain all non-volatile, encodable phase levels of light that can be transmitted by the phase change optical synaptic device, as well as all volatile modulation levels of each phase state and a fully crystalline state level with high light attenuation. Map all device transmittance levels (except the fully crystalline state level) to the weight range required by the convolutional neural network model.

[0113] Step S2: Construct a convolutional neural network model, a coefficient generation model, and the size selection model in the computer module; according to all available weight values obtained in step S1, input the training data set into the computer module for training, and the parameters to be trained include the parameters of the coefficient generation model, the parameters of the size selection model, the correspondence between the coefficient generation result and the volatile modulation excitation, the static convolution kernel value of the dynamic convolution module, and other parameters of the convolutional neural network model; adopt quantized training to train, and use optimization algorithms such as gradient descent to continuously adjust and optimize the weights in the network through forward propagation and back propagation processes to minimize the loss function; coordinately train all parameters, where the weight value of the static convolution kernel is selected at the non-volatile phase level.

[0114] Step S3: After the training is completed, the number of phase change optical synapse devices in the dynamic convolution module is determined according to S2, and the phase change optical synapse devices are used to implement the convolution operation weight encoding in the convolutional neural network model, and one phase change optical synapse device corresponds to one convolution kernel weight; the static convolution kernel weight value obtained by training in step S2 is loaded into the phase change material unit of the dynamic convolution module in a non-volatile modulation method such as electrically controlled heating phase change, and the weight value is encoded into the corresponding non-volatile phase state of the corresponding phase change optical synapse device, and the weight size is characterized by light transmittance.

[0115] Step S4: Determine the number of optical signals and the corresponding hardware quantity of the light source module and the photoelectric conversion module, and then determine the number of multiplexers and balanced photodetectors; the light source module simultaneously outputs m optical signals encoding input data information, where m depends on the size of the convolution kernel. Specifically, when the convolution kernel size is [3×3], m=9; the photoelectric conversion module simultaneously has p optical signals for photoelectric conversion, where p depends on the number of convolution kernels; and then determine the quantity of hardware required for the light source module and the photoelectric conversion module.

[0116] Step S5: Deploy the corresponding light source module and photoelectric conversion module, map the neural network onto the chip, establish the volatile modulation relationship between the coefficient generation model and the dynamic convolution module, establish the high light attenuation modulation relationship between the size selection model and the dynamic convolution module, set the control logic of the computer module, and form a photonic convolutional neural network system.

[0117] In summary, the present invention combines the non-volatile modulation of the phase state of phase change materials with the volatile modulation to jointly control the transmission of light in the waveguide and construct a dynamic photon convolution kernel; through the coordinated modulation of the two modulation methods, the flexibility of the phase change photon convolution kernel is improved, so that the photon convolution neural network system can be dynamically adjusted according to different inputs, which helps to promote the application of the photon convolution neural network system in larger-scale and more complex task scenarios and improve performance.

[0118] The above embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, and all such variations and improvements fall within the scope of protection of the present invention.

[0119] It will be easily understood by those skilled in the art that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A photonic convolutional neural network system, characterized in that: include: Dynamic convolution kernel modules that correspond one-to-one to each convolution layer in the pre-trained convolutional neural network model; The dynamic convolution kernel module includes: a photonic synaptic device array; the array stores pre-trained convolution kernel weights in the corresponding convolution layer; wherein the photonic synaptic devices in the array are photonic synaptic devices based on phase change materials; the convolution kernel weights are encoded into the corresponding non-volatile phase states of the photonic synaptic devices through non-volatile modulation, thereby achieving storage; one photonic synaptic device stores one convolution kernel weight; and the light transmittance of the photonic synaptic devices in different phase states is different. The dynamic convolution kernel module is used to dynamically adjust the convolution kernel weights stored in the array before performing the convolution operation: receiving the dynamic coefficients corresponding to the convolution kernel weights stored in the array, and applying a pulse signal corresponding to the corresponding dynamic coefficient to each photonic synapse device for volatile modulation; The dynamic convolution kernel module is further configured to receive, during a convolution operation, a continuous signal light carrying data information to be convolved, the continuous signal light being attenuated by the array, and outputting the attenuated signal light; the intensity of the output signal light being the result of the current convolution operation; The dynamic coefficient is generated by inputting the data to be convolutionally operated into the corresponding pre-trained coefficient generation model; one dynamic convolution kernel module corresponds to one coefficient generation model, and the coefficient generation model is a deep learning model; The weights in the same convolution kernel are stored in the same column of the array; The dynamic convolution kernel module is further configured to dynamically adjust the size of each convolution kernel stored in the array before performing a convolution operation and after dynamically adjusting the convolution kernel weights stored in the array: selecting a number of photonic synaptic devices from each column of the array, maintaining the phase state of the selected photonic synaptic devices unchanged, and modulating the phase state of the unselected photonic synaptic devices to a fully crystalline state through a non-volatile modulation method, so as to adjust the size of the convolution kernel stored in the column to a corresponding optimal size; Among them, a dynamic convolution kernel module also corresponds to a size selection model, and the size selection model is a deep learning model; The optimal size of each convolution kernel stored in the array is generated by inputting the data to be convolved into the corresponding size selection module; the size selection model is used to extract the features of the data to be convolved and map them to the optimal size of each convolution kernel stored in the array within the corresponding dynamic convolution kernel module; The size selection model includes: a cascaded feature extraction module and a classifier; the feature extraction module includes: one or more of an edge feature extraction unit, a texture feature extraction unit, and a smoothness feature extraction unit, for extracting one or more of the edge features, texture features, and smoothness features of the data to be convolved.

2. The photonic convolutional neural network system according to claim 1, characterized in that The coefficient generation model includes: a plurality of cascaded fully connected layers, an activation layer arranged between two adjacent fully connected layers, and a pooling layer connected before the first fully connected layer.

3. The photonic convolutional neural network system according to claim 1, wherein: The training methods for the coefficient generation models corresponding to each dynamic convolution kernel module include: The coefficient generation model is introduced during the training process of the convolutional neural network model, and the coefficient generation models corresponding to each dynamic convolution kernel module correspond one-to-one to each convolution layer in the convolutional neural network model; in the forward propagation process, before performing the convolution operation, each convolution layer generates the dynamic coefficients corresponding to the weights of each convolution kernel in the convolution layer based on the data input to the convolution layer through the corresponding coefficient generation model, and dynamically adjusts the corresponding convolution kernel weights in the convolution layer based on the obtained dynamic coefficients; in the backward propagation process, the parameters in the convolutional neural network model and each coefficient generation model are adjusted at the same time.

4. The photonic convolutional neural network system according to claim 1, wherein: The classifier is a fully connected conditional network, wherein the fully connected conditional network includes a plurality of cascaded fully connected layers, an activation layer arranged between two adjacent fully connected layers, and a softmax layer connected after the last fully connected layer.

5. The photonic convolutional neural network system according to claim 1, wherein: The training methods for the coefficient generation model and size selection model corresponding to each dynamic convolution kernel module include: The coefficient generation model and the size selection model are introduced during the training process of the convolutional neural network model, wherein the coefficient generation model corresponding to each dynamic convolution kernel module corresponds one-to-one to each convolution layer in the convolutional neural network model; and the size selection model corresponding to each dynamic convolution kernel module corresponds one-to-one to each convolution layer in the convolutional neural network model; During the forward propagation process, before performing the convolution operation, each convolution layer generates the dynamic coefficients corresponding to the weights of each convolution kernel in the convolution layer based on the input data of the convolution layer through the corresponding coefficient generation model, and dynamically adjusts the corresponding convolution kernel weights in the convolution layer based on the obtained dynamic coefficients; and generates the optimal size of each convolution kernel in the convolution layer based on the input data of the convolution layer through the corresponding size selection model, so as to dynamically adjust the size of each convolution kernel in the convolution layer; During the back-propagation process, the parameters in the convolutional neural network model, the coefficient generation models, and the size selection models are adjusted simultaneously.

6. The photonic convolutional neural network system according to any one of claims 1 to 3, wherein: Also includes: The light source module is used to load the data to be currently convolved onto the optical signal using an optical signal modulator to obtain a continuous signal light carrying the data information to be currently convolved.

7. The photonic convolutional neural network system according to any one of claims 1 to 3, wherein: Also includes: Photoelectric conversion module, nonlinear activation module and fully connected layer module; The photoelectric conversion module is used to convert the signal light output by the dynamic convolution kernel module into an electrical signal, and output it to the nonlinear activation module; The nonlinear activation module is used to perform nonlinear activation processing on the electrical signal input by the photoelectric conversion module; The fully connected layer module is used to calculate the nonlinear activation processing result output by the nonlinear activation module for the last time to obtain the final output result.

8. The photonic convolutional neural network system according to any one of claims 1 to 3, wherein: The photonic synapse device comprises: a substrate, a waveguide layer, a phase change material layer, a heating layer, a covering layer, and an electrode layer acting on the heating layer, which are arranged in sequence from bottom to top.

Images