Intelligent optical computing nonlinear convolution chip system, method and architecture

Abstract

This disclosure relates to the field of optical computing technology, and in particular to an intelligent optical computing nonlinear convolutional chip system, method, and architecture, comprising: an input spatiotemporal transformation module for processing an input one-dimensional matrix to generate a complex-weighted extended channel matrix; an input temporal encoding and nonlinear activation module for performing nonlinear activation and first information delay on the complex-weighted extended channel matrix to obtain a matrix; a one-dimensional convolution processing module for performing matrix multiplication on the first matrix to obtain a second matrix; an output temporal encoding and nonlinear activation module for performing nonlinear activation and second information delay on the second matrix to obtain a third matrix; and an output spatiotemporal transformation module for performing complex-weighted channel fusion operation on the third matrix to obtain an output one-dimensional matrix. This disclosure avoids the use of photoelectric conversion and off-chip devices, realizing an all-optical nonlinear convolutional system that can be cascaded to form a large-scale deep all-optical network with high integration, high speed, and strong scalability.

Description

Technical Field

[0001] This disclosure relates to the field of optical computing technology, and in particular to an intelligent optical computing nonlinear convolution chip system, method and architecture. Background Technology

[0002] With the rapid development of artificial intelligence and scientific computing, the complexity and scale of computing demands are constantly increasing. However, existing electronic computing hardware (such as CPUs, GPUs, FPGAs, and ASICs) will struggle to handle significant computational burdens, facing saturation in computing speed and energy efficiency. In the context of the post-Moore's Law era and the convergence of Artificial General Intelligence (AGI), realizing the high computing potential of computing architectures has become a persistent goal of high-performance computing research. Light possesses the advantages of high throughput and low latency during propagation. Based on this, photonic computing units can continuously process input data with high throughput as it flows through the processor, enabling vowel recognition, serial data classification, and time series prediction. Summary of the Invention

[0003] This disclosure aims to at least partially address one of the technical problems in the related art.

[0004] Therefore, the first objective of this disclosure is to propose an intelligent optical computing nonlinear convolutional chip system. This system achieves nonlinear activation and information delay through an integrated first and second micro-ring resonator array. Furthermore, it transforms the input one-dimensional matrix into matrix-vector multiplication by interleaving time and space dimensions using a first and second phase modulator array, avoiding photoelectric conversion and the use of off-chip devices. This realizes an all-optical nonlinear convolutional system that can be directly cascaded to form large-scale deep all-optical networks. It boasts advantages such as high integration, high speed, and strong scalability, accelerating the development of machine vision towards sub-nanosecond levels. Simultaneously, the intelligent optical computing nonlinear convolutional chip system constructs complex temporal convolutional neural networks, exhibiting superior performance in image classification, high-speed dynamic video classification, and human motion generation tasks.

[0005] The second objective of this disclosure is to propose an image classification method.

[0006] To achieve the above objectives, a first aspect of this disclosure proposes an intelligent optical computing nonlinear convolution chip system. The system includes an input spatiotemporal transformation module, an input temporal coding and nonlinear activation module, a one-dimensional convolution processing module, an output temporal coding and nonlinear activation module, and an output spatiotemporal transformation module.

[0007] The input spatiotemporal conversion module is used to obtain the input one-dimensional matrix to be analyzed, and to process the input one-dimensional matrix through the first phase modulator array to generate the complex weighted extended channel matrix corresponding to the input one-dimensional matrix.

[0008] The input timing encoding and nonlinear activation module is used to perform nonlinear activation and first information delay on the complex weighted extended channel matrix through the first micro-ring resonator array to obtain the corresponding first matrix;

[0009] The one-dimensional convolution processing module is used to perform matrix multiplication on the first matrix through the second phase modulator array to obtain the corresponding second matrix.

[0010] The output timing encoding and nonlinear activation module is used to perform nonlinear activation and second information delay on the second matrix through the second microring resonator array to obtain the corresponding third matrix;

[0011] The output spatiotemporal conversion module is used to perform a complex-weighted channel fusion operation on the third matrix through the first phase modulator array to obtain an output one-dimensional matrix.

[0012] Optionally, the first phase modulator array is implemented using a Mach-Zehnder interferometer to determine the amplitude modulation coefficient and phase modulation coefficient of each channel for light.

[0013] Optionally, the first microring resonator array completes information delay through group delay effect and achieves all-optical nonlinearity through Kerr nonlinearity and free carrier dispersion effect.

[0014] Optionally, the intelligent optical computing nonlinear convolutional chip system is used for image classification, high-speed dynamic video classification, and human motion generation tasks.

[0015] To achieve the above objectives, a second aspect of this disclosure provides an image classification method, including:

[0016] Acquire the image that needs to be analyzed;

[0017] The image is preprocessed to obtain a corresponding one-dimensional image matrix;

[0018] The one-dimensional image matrix is ​​processed by a target feature extraction model to obtain a corresponding target feature matrix. The target feature extraction model consists of at least one intelligent optical computing nonlinear convolution chip system.

[0019] Based on the target feature matrix, the classification result of the image is determined.

[0020] Optionally, before extracting features from the one-dimensional image matrix using a target feature extraction model to obtain the corresponding target feature matrix, the method further includes:

[0021] Determine the number of layers in the convolutional architecture and the number of image classifications;

[0022] Based on the number of layers in the convolutional architecture, the intelligent optical computing nonlinear convolutional chip system is connected in series to obtain each layer of the convolutional network;

[0023] Based on the number of image classifications, each layer of the convolutional network is connected in parallel to obtain an initial feature extraction model.

[0024] Optionally, the step of extracting features from the one-dimensional image matrix using a target feature extraction model to obtain the corresponding target feature matrix includes:

[0025] The one-dimensional image matrix is ​​input into each layer of the convolutional network, and then passed through the intelligent optical computing nonlinear convolutional chip system connected in series in each layer of the convolutional network to obtain the one-dimensional feature matrix of each layer of the convolutional network.

[0026] The one-dimensional feature matrices are concatenated to obtain the corresponding target feature matrix.

[0027] Optionally, determining the classification result of the image based on the target feature matrix includes: determining the classification result of the image by detecting and accumulating the output light intensity using a photodetector based on the target feature matrix.

[0028] Another object of the present invention is to provide an image classification device, characterized in that it comprises:

[0029] The acquisition module is used to acquire the images that need to be analyzed.

[0030] The data processing module is used to preprocess the image to obtain a corresponding one-dimensional image matrix;

[0031] The feature extraction module is used to extract features from the one-dimensional image matrix through the target feature extraction model to obtain the corresponding target feature matrix, wherein the target feature extraction model is composed of at least one intelligent optical computing nonlinear convolution chip system;

[0032] A classification module is used to determine the classification result of the image based on the target feature matrix.

[0033] In summary, the intelligent optical computing nonlinear convolutional chip system, method, and architecture provided in this disclosure achieve nonlinear activation and information delay through an integrated first and second micro-ring resonator array. Furthermore, by using a first and second phase modulator array, the input one-dimensional matrix is ​​transformed into matrix-vector multiplication through interleaved time and spatial dimensions, avoiding photoelectric conversion and the use of off-chip devices. This realizes an all-optical nonlinear convolutional system that can be directly cascaded to form large-scale deep all-optical networks. It boasts advantages such as high integration, high speed, and strong scalability, accelerating the development of machine vision towards sub-nanosecond levels. Simultaneously, the intelligent optical computing nonlinear convolutional chip system constructs complex temporal convolutional neural networks, exhibiting superior performance in image classification, high-speed dynamic video classification, and human motion generation tasks.

[0034] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0035] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0036] Figure 1 This is a schematic diagram of the structure of an intelligent optical computing nonlinear convolution chip system provided in an embodiment of the present disclosure;

[0037] Figure 2 This is a schematic diagram of the optical structure of an intelligent optical computing nonlinear convolution chip system provided in an embodiment of this disclosure;

[0038] Figure 3 A schematic diagram illustrating the simulation and experimental output and depth detection results of a nonlinear convolutional chip system based on intelligent optical computing, provided in an embodiment of this disclosure.

[0039] Figure 4 A network structure diagram for human action generation provided in this embodiment of the disclosure;

[0040] Figure 5 This is a schematic diagram of the architecture of an operational intelligent optical computing nonlinear convolution chip system provided in an embodiment of this disclosure;

[0041] Figure 6 This is a schematic flowchart of an image classification method proposed in an embodiment of this disclosure;

[0042] Figure 7 This is a schematic diagram of an image classification method proposed in an embodiment of this disclosure;

[0043] Figure 8 This is a schematic diagram of the structure of an image classification device provided in an embodiment of the present disclosure. Detailed Implementation

[0044] Embodiments of this disclosure are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0045] Currently, photonic processing units still face challenges in achieving integrated and scalable nonlinear activation, limiting the potential of large-scale photonic neural networks and complex artificial intelligence tasks. For example, optical nonlinearity can be achieved through laser-cooled atoms, photorefractive crystals, and light scattering, but these free-space solutions are difficult to integrate onto chips, thus weakening their compatibility, scalability, and stability. On the other hand, on-chip nonlinear activation mainly relies on photoelectric conversion and transimpedance amplifiers, requiring heterogeneous integration with CMOS electronics, leading to low energy efficiency, redundant delays, and increased system complexity. Therefore, all-optically integrated nonlinear activation is crucial for next-generation photonic neural networks.

[0046] The present disclosure will now be described in detail with reference to specific embodiments.

[0047] Figure 1 This is a schematic diagram of the structure of an intelligent optical computing nonlinear convolution chip system provided in an embodiment of this disclosure. Figure 1 As shown, the model includes an input spatiotemporal transformation module 101, an input temporal coding and nonlinear activation module 102, a one-dimensional convolution processing module 103, an output temporal coding and nonlinear activation module 104, and an output spatiotemporal transformation module 105, wherein...

[0048] The input spatiotemporal conversion module 101 is used to obtain the input one-dimensional matrix to be analyzed, and to process the input one-dimensional matrix through the first phase modulator array to generate the complex weighted extended channel matrix corresponding to the input one-dimensional matrix.

[0049] The input timing encoding and nonlinear activation module 102 is used to perform nonlinear activation and first information delay on the complex weighted extended channel matrix through the first micro-ring resonator array to obtain the corresponding first matrix;

[0050] The one-dimensional convolution processing module 103 is used to perform matrix multiplication on the first matrix through the second phase modulator array to obtain the corresponding second matrix.

[0051] The output timing encoding and nonlinear activation module 104 is used to perform nonlinear activation on the second matrix and delay the second information to obtain the corresponding third matrix through the second micro-ring resonator array;

[0052] The output time-space conversion module 105 is used to perform complex-weighted channel fusion operation on the third matrix through the first phase modulator array to obtain an output one-dimensional matrix.

[0053] In one embodiment of this disclosure, the above-described intelligent optical computing nonlinear convolutional chip system can be applied to a variety of tasks, including but not limited to image classification, high-speed dynamic video classification, and human motion generation tasks.

[0054] In one embodiment of this disclosure, the input one-dimensional matrix to be analyzed can be an input one-dimensional matrix obtained after data preprocessing of input data for different tasks. For example, in one embodiment of this disclosure, assuming that the intelligent optical computing nonlinear convolutional chip system is suitable for image classification tasks, the input one-dimensional matrix obtained is the input one-dimensional matrix of the image obtained through data processing.

[0055] Furthermore, in one embodiment of this disclosure, the aforementioned first phase modulator array can be implemented using a Mach-Zehnder interferometer to determine the amplitude modulation coefficient and phase modulation coefficient of each channel for light. Also, in one embodiment of the disclosure, the aforementioned input spatiotemporal conversion module 101 can use the first phase modulator array to weight the channel information of the input one-dimensional matrix and increase or decrease the number of channels to achieve spatial dimension changes in the information.

[0056] In one embodiment of this disclosure, the first microring resonator array can be a tunable microring resonator. In another embodiment, the first microring resonator array achieves information delay through group delay effect and realizes all-optical nonlinearity through Kerr nonlinearity and free carrier dispersion effect. Furthermore, in another embodiment, the first microring resonator array can use a thermoelectric modulator to change the reflectivity of the microrings, thereby altering the free carrier dispersion effect and group delay effect of the microrings.

[0057] Furthermore, in one embodiment of this disclosure, the number of activated microrings in the first microring resonator array and the second microring resonator array may be different. Specifically, in one embodiment of this disclosure, the number of activated microrings corresponding to the first microring resonator array may be 12, 8, 4, or 0; and the number of activated microrings corresponding to the second microring resonator array may be 3, 2, 1, or 0.

[0058] Furthermore, in one embodiment of this disclosure, the all-optical nonlinearity achieved by the first microring resonator array avoids photoelectric conversion and the use of external devices, thereby enabling the intelligent optical computing nonlinear convolution chip system to constitute an intelligent optical computing nonlinear convolution chip.

[0059] Figure 2This is a schematic diagram of the optical structure of an intelligent optical computing nonlinear convolution chip system provided in an embodiment of this disclosure.

[0060] Figure 3 This diagram illustrates the simulation and experimental outputs and depth detection results of a nonlinear convolutional chip system based on intelligent optical computing, as provided in an embodiment of this disclosure. Figure 3 a is an optical micrograph of the entire chip; Figure 3 b is an optical micrograph of a microring resonator (MRR) and a Mach-Zehnder interferometer (MZI); Figure 3 c represents the optical time delay using a pulse signal as input when 2 or 4 MRRs are activated (corresponding to the left and right panels respectively), with the shaded area indicating the delay time; Figure 3 d represents the time delay when different numbers of MRRs are activated; Figure 3 e represents the relationship between experimental intensity measurement and phase change and input intensity when 2 or 4 MRRs are activated (left and middle panels). Figure 3 f represents the phase nonlinearity of MRR with different numbers of activations.

[0061] In one embodiment of this disclosure, the aforementioned intelligent optical computing nonlinear convolutional chip system establishes an all-optical computing system with an all-optical nonlinear structure and high-order convolution, exhibiting superior performance compared to electronic networks of the same structure (taking a fully convolutional structure network with ReLU nonlinearity as an example). In one embodiment of this disclosure, the aforementioned intelligent optical computing nonlinear convolutional chip system can operate normally under long-distance (up to 6 meters), low-power (minimum 0.3µW), high-speed (up to 600Hz), and high-transmittance conditions, demonstrating the performance advantages of the intelligent optical computing nonlinear convolutional chip system compared to traditional depth sensing methods such as LiDAR. The reconfigurable neuron activation constructed by the intelligent optical computing nonlinear convolutional chip system results in a latency as low as 0.257 nanoseconds, achieving a computing power of 4.96 TOPS on a single chip.

[0062] Furthermore, in one embodiment of this disclosure, the TPPU network composed of the aforementioned intelligent optical computing nonlinear convolutional chip system was experimentally evaluated on an image classification task, achieving an accuracy of 90.1% (compared to 84.2% without nonlinearity). In another embodiment of this disclosure, the extended TPPU network demonstrated the ability to perform more complex human motion generation, with an FID (Fréchet distance) fidelity index of 8.424 (compared to 14.77 without nonlinearity).

[0063] Furthermore, in one embodiment of this disclosure, the intelligent optical computing nonlinear convolutional chip system can be directly cascaded to form a large-scale optical network. In deep networks, fiber amplifiers can be used for reinforcement connections to enhance light intensity, enabling the deep network to function properly.

[0064] Example, Figure 4 A network structure diagram for human action generation provided in this disclosure embodiment, such as... Figure 4 As shown, the human action generation network constructs an encoder-decoder architecture based on a general stack composed of multiple intelligent optical computing nonlinear convolutional chip systems. This encoder-decoder architecture is capable of extracting features between elements in a sequence when there is a complex mapping between the input and output sequences. Figure 4 As shown, each frame of the human action sequence contains the three-dimensional Cartesian coordinates of 20 key joints of the human body. During the generation process, the joint position information of the four frames before and after the current frame is used as input. Furthermore, during the training of the aforementioned human action generation network, the generated action sequence is optimized using a Markov chain Monte Carlo method, and the feature extraction quality of the generated action is evaluated using Fraser distance (FID) and kernel distance (KID). The smaller the value, the higher the similarity between the features of the generated action and the features of the original action.

[0065] In one embodiment of this disclosure, during the training of the aforementioned human action generation network, the corresponding loss function can be defined as:

[0066]

[0067] in Indicates the first movement The true values ​​of the spatial Cartesian coordinates of each node This represents the predicted network coordinates, where n is the total number of joints. This number can be set as needed; for example, n=20.

[0068] Figure 5 This is a schematic diagram of the architecture of an operational intelligent optical computing nonlinear convolution chip system provided in an embodiment of this disclosure. Figure 5As shown, the architecture uses a thermoelectric cooler (TED200C, Thorlabs) with a slow proportional-integral-derivative feedback loop to stabilize the chip temperature, with electrical probes contacting the micro-heating elements. To simultaneously control the states of the modulated microring resonator (MRR) and phase adjuster, the architecture employs twelve 6-channel isolated arbitrary waveform generators (PXI-7961, CHNNI Instruments) as inputs to the device's photothermal controller. The beam is generated by a tunable solid-state laser (CoBriteDX–Tunable Laser) operating at a wavelength of 1550 nm. The beam is coupled via single-mode fiber to a lithium niobate electro-optic modulator (LN81S-FC, Thorlabs) to modulate the beam intensity. An arbitrary waveform generator (AWG70001B, Tektronix) converts the data into an electrical waveform of a specific frequency. The optical signal is polarized by a linear polarizer (MPC320, Thorlabs) and coupled to the chip's input channel as the system input signal. The chip's output channel is connected to an InGaAs photodetector (DET08CFC / M, Thorlabs) to convert the output light intensity into an electrical signal, which is then captured by a high-speed oscilloscope (MSO64B, Tektronix).

[0069] The intelligent optical computing nonlinear convolutional chip system of this disclosure achieves nonlinear activation and information delay through an integrated first microring resonator array and a second microring resonator array. It transforms the input one-dimensional matrix into matrix-vector multiplication by interleaving time and space dimensions through a first phase modulator array and a second phase modulator array, avoiding photoelectric conversion and the use of off-chip devices. This realizes an all-optical nonlinear convolutional system that can be directly cascaded to form large-scale deep all-optical networks. It boasts advantages such as high integration, high speed, and strong scalability, accelerating the development of machine vision towards sub-nanosecond levels. Furthermore, the intelligent optical computing nonlinear convolutional chip system constructs complex temporal convolutional neural networks, exhibiting superior performance in image classification, high-speed dynamic video classification, and human motion generation tasks.

[0070] To achieve the above embodiments, Figure 6 This disclosure also proposes an image classification method, which may include the following steps:

[0071] Step 601: Obtain the image to be analyzed;

[0072] Step 602: Perform data preprocessing on the image to obtain the corresponding one-dimensional image matrix;

[0073] Step 603: Extract features from the one-dimensional image matrix using a target feature extraction model to obtain the corresponding target feature matrix;

[0074] Step 604: Determine the classification result of the image based on the target feature matrix.

[0075] In one embodiment of this disclosure, after acquiring the image to be analyzed, the image can be preprocessed to obtain a corresponding one-dimensional image matrix. Specifically, in one embodiment of this disclosure, the multi-dimensional matrix corresponding to the image can be rearranged to obtain the corresponding one-dimensional image matrix.

[0076] In one embodiment of this disclosure, the target feature extraction model consists of at least one intelligent optical computing nonlinear convolution chip system.

[0077] In one embodiment of this disclosure, before extracting features from the one-dimensional image matrix using a target feature extraction model to obtain the corresponding target feature matrix, the above method may further include the following steps:

[0078] Step 1: Determine the number of layers in the convolutional architecture and the number of image classifications;

[0079] Step 2: Based on the number of layers in the convolutional architecture, the intelligent optical computing nonlinear convolutional chip system is connected in series to obtain each layer of the convolutional network;

[0080] Step 3: Based on the number of image classifications, connect each convolutional network layer in parallel to obtain the initial feature extraction model.

[0081] In one embodiment of this disclosure, the number of layers in the convolutional architecture and the number of image classifications can be determined through user input or human experience. In one embodiment, the target feature extraction model can consist of an N-layer fully convolutional architecture, with the network having M channels for an M-classification task. Each layer contains N×M×4 convolutional kernels. For example, in one embodiment of this disclosure, the convolutional architecture has 5 layers, and the number of image classifications is 4.

[0082] In one embodiment of this disclosure, after obtaining the initial feature extraction model through the above steps, the initial feature extraction model can be trained, and the trained feature extraction model can be determined as the target feature extraction model. In one embodiment of this disclosure, the method for training the initial feature extraction model is the same as the existing training method, and will not be described in detail here.

[0083] In one embodiment of this disclosure, the loss function in the above training process can be:

[0084] ,

[0085] ,

[0086] ,

[0087] in, Represents the cross-entropy loss function. Represents the energy distribution loss function. Represents the number of categories, It is an indicator function (0 or 1), when the sample The true category is The value is 1 when the condition is met, and 0 otherwise; prediction probability Observed values Category The predicted probability, and Indicates the ratio of predicted probability distributions. This represents the number of samples.

[0088] In one embodiment of this disclosure, the cross-entropy loss function and the energy distribution loss function in the above-mentioned loss functions can guide the optimization of the output energy distribution, ensuring that the optical experimental results are consistent with the simulation results.

[0089] Furthermore, in one embodiment of this disclosure, the method for extracting features from a one-dimensional image matrix using a target feature extraction model to obtain a corresponding target feature matrix may include the following steps:

[0090] Step 6031: Input the one-dimensional image matrix into each layer of the convolutional network, and pass it sequentially through the intelligent optical computing nonlinear convolutional chip system connected in series in each layer of the convolutional network to obtain the one-dimensional feature matrix of each layer of the convolutional network.

[0091] Step 6032: Concatenate the one-dimensional feature matrices to obtain the corresponding target feature matrix.

[0092] Furthermore, in one embodiment of this disclosure, after obtaining the target feature matrix through the above steps, the classification result of the image can be determined based on the target feature matrix.

[0093] Specifically, in one embodiment of this disclosure, the method for determining the classification result of an image based on a target feature matrix may include: determining the classification result of the image by detecting and accumulating the output light intensity using a photodetector based on the target feature matrix.

[0094] Specifically, in one embodiment of this disclosure, each channel corresponds to a classification category. Within a specified number of time steps, the output light intensity of each channel is accumulated to obtain the accumulated light intensity of each channel. The channel corresponding to the maximum value of the accumulated light intensity reflects the category of the image.

[0095] Based on the above description Figure 7 This is a schematic diagram of an image classification method proposed in an embodiment of this disclosure. Figure 7 As shown,

[0096] The input image is preprocessed by rearranging the original data from 4×4 to obtain a 1×16 one-dimensional image matrix. Then, each of the four convolutional networks in the target feature extraction model is used to obtain a 1×16 one-dimensional feature matrix corresponding to each convolutional network. The one-dimensional feature matrices of the four convolutional networks are concatenated to obtain a 4×1×16 target feature matrix. Based on the target feature matrix, the classification result of the image is determined by detecting and accumulating the output light intensity using a photodetector. The feature extraction model includes two intelligent optical computing nonlinear convolutional chip systems connected in series to obtain each convolutional network, and consists of four convolutional networks connected in parallel.

[0097] In this embodiment of the disclosure, the above-mentioned image classification method, through the target feature extraction model composed of an intelligent optical computing nonlinear convolution chip system, possesses strong nonlinear performance and representation capabilities, thereby making the image classification results more accurate and improving the image classification accuracy.

[0098] To achieve the above embodiments, Figure 8 This disclosure also proposes an image classification apparatus, which may include:

[0099] The acquisition module 801 is used to acquire the image to be analyzed;

[0100] The data processing module 802 is used to preprocess the image data to obtain the corresponding one-dimensional image matrix;

[0101] The feature extraction module 803 is used to extract features from a one-dimensional image matrix through a target feature extraction model to obtain a corresponding target feature matrix. The target feature extraction model consists of at least one intelligent optical computing nonlinear convolution chip system.

[0102] The classification module 804 is used to determine the classification result of the image based on the target feature matrix.

[0103] In one embodiment of this disclosure, the array of intelligent optical computing nonlinear convolutional chip systems is divided into different groups, and the different groups of intelligent optical computing nonlinear convolutional chip systems correspond to different wavelengths in the preset incident light.

[0104] In one embodiment of this disclosure, the above-described apparatus is further used for:

[0105] Determine the number of layers in the convolutional architecture and the number of image classifications;

[0106] Based on the number of layers in the convolutional architecture, the intelligent optical computing nonlinear convolutional chip system is connected in series to obtain each layer of the convolutional network;

[0107] Based on the number of image classifications, each convolutional network layer is connected in parallel to obtain the initial feature extraction model.

[0108] In one embodiment of this disclosure, the feature extraction module 803 is specifically used for:

[0109] The one-dimensional image matrix is ​​input into each layer of the convolutional network, and then passed through the intelligent optical computing nonlinear convolutional chip system connected in series in each layer of the convolutional network to obtain the one-dimensional feature matrix of each layer of the convolutional network.

[0110] The one-dimensional feature matrices are concatenated to obtain the corresponding target feature matrix.

[0111] In one embodiment of this disclosure, the classification module 804 is specifically used for:

[0112] Based on the target feature matrix, the classification result of the image is determined by detecting and accumulating the output light intensity using a photodetector.

[0113] In this embodiment of the disclosure, the image classification system described above, through the target feature extraction model composed of an intelligent optical computing nonlinear convolution chip system, possesses strong nonlinear performance and representation capabilities, thereby making the image classification results more accurate and improving the image classification accuracy.

[0114] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this disclosure all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0115] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0116] This disclosure is intended to provide implementation schemes for users to selectively prevent the use or access to their personal information data. Specifically, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.

[0117] The acquisition, transmission, storage, use, and processing of data in this disclosed technical solution all comply with the relevant provisions of national laws and regulations.

[0118] It should be noted that in the embodiments disclosed herein, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary and are intended only to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used such solutions.

[0119] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0120] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0121] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0122] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0123] It should be understood that various parts of this disclosure can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0124] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0125] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0126] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A smart optical computing nonlinear convolution chip system, characterized in that, The system includes an input spatiotemporal transformation module, an input temporal coding and nonlinear activation module, a one-dimensional convolution processing module, an output temporal coding and nonlinear activation module, and an output spatiotemporal transformation module, wherein... The input spatiotemporal conversion module is used to obtain the input one-dimensional matrix to be analyzed, and to process the input one-dimensional matrix through the first phase modulator array to generate the complex weighted extended channel matrix corresponding to the input one-dimensional matrix. The input timing encoding and nonlinear activation module is used to perform nonlinear activation and first information delay on the complex weighted extended channel matrix through the first micro-ring resonator array to obtain the corresponding first matrix; The one-dimensional convolution processing module is used to perform matrix multiplication on the first matrix through the second phase modulator array to obtain the corresponding second matrix. The output timing encoding and nonlinear activation module is used to perform nonlinear activation and second information delay on the second matrix through the second microring resonator array to obtain the corresponding third matrix; The output spatiotemporal conversion module is used to perform a complex-weighted channel fusion operation on the third matrix through the first phase modulator array to obtain an output one-dimensional matrix; The first microring resonator array is a tunable microring resonator. The first microring resonator array completes information delay through group delay effect and realizes all-optical nonlinearity through Kerr nonlinearity and free carrier dispersion effect. The number of activated microrings corresponding to the first microring resonator array is 12, 8, 4, 0; the number of activated microrings corresponding to the second microring resonator array is 3, 2, 1, 0. The aforementioned intelligent optical computing nonlinear convolutional chip system is used for image classification, high-speed dynamic video classification, and human motion generation tasks.

2. The system according to claim 1, characterized in that, The first phase modulator array is implemented using a Mach-Zehnder interferometer to determine the amplitude modulation coefficient and phase modulation coefficient of each channel for light.

3. An image classification method, characterized in that, include: Acquire the image that needs to be analyzed; The image is preprocessed to obtain a corresponding one-dimensional image matrix; The one-dimensional image matrix is ​​subjected to feature extraction through a target feature extraction model to obtain a corresponding target feature matrix, wherein the target feature extraction model is composed of at least one intelligent optical computing nonlinear convolution chip system as described in any one of claims 1 to 2; Based on the target feature matrix, the classification result of the image is determined.

4. The method according to claim 3, characterized in that, Before extracting features from the one-dimensional image matrix using a target feature extraction model to obtain the corresponding target feature matrix, the method further includes: Determine the number of layers in the convolutional architecture and the number of image classifications; Based on the number of layers in the convolutional architecture, the intelligent optical computing nonlinear convolutional chip system is connected in series to obtain each layer of the convolutional network; Based on the number of image classifications, each layer of the convolutional network is connected in parallel to obtain an initial feature extraction model.

5. The method according to claim 4, characterized in that, The step of extracting features from the one-dimensional image matrix using a target feature extraction model to obtain the corresponding target feature matrix includes: The one-dimensional image matrix is ​​input into each layer of the convolutional network, and then passed through the intelligent optical computing nonlinear convolutional chip system connected in series in each layer of the convolutional network to obtain the one-dimensional feature matrix of each layer of the convolutional network. The one-dimensional feature matrices are concatenated to obtain the corresponding target feature matrix.

6. The method according to claim 3, characterized in that, The step of determining the classification result of the image based on the target feature matrix includes: determining the classification result of the image by detecting and accumulating the output light intensity using a photodetector based on the target feature matrix.

7. An image classification device, characterized in that, include: The acquisition module is used to acquire the images that need to be analyzed. The data processing module is used to preprocess the image to obtain a corresponding one-dimensional image matrix; The feature extraction module is used to extract features from the one-dimensional image matrix through a target feature extraction model to obtain a corresponding target feature matrix, wherein the target feature extraction model is composed of at least one intelligent optical computing nonlinear convolution chip system as described in any one of claims 1 to 2; A classification module is used to determine the classification result of the image based on the target feature matrix.

8. The apparatus according to claim 7, characterized in that, The device is also used for: Determine the number of layers in the convolutional architecture and the number of image classifications; Based on the number of layers in the convolutional architecture, the intelligent optical computing nonlinear convolutional chip system is connected in series to obtain each layer of the convolutional network; Based on the number of image classifications, each layer of the convolutional network is connected in parallel to obtain an initial feature extraction model.

Images