Intelligent optical computing non-linear convolution chip system, method and architecture

Abstract

The invention relates to the technical field of optical computing, in particular to an intelligent optical computing nonlinear convolution chip system, method and architecture, and the system comprises an input time-space conversion module which is used for processing an input one-dimensional matrix to generate a complex weighted expansion channel matrix; the input time sequence coding and nonlinear activation module is used for performing nonlinear activation and first information delay on the complex weighted expansion channel matrix to obtain a matrix; the one-dimensional convolution processing module is used for performing matrix multiplication on the first matrix to obtain a second matrix; the output time sequence coding and nonlinear activation module is used for performing nonlinear activation and second information delay on the second matrix to obtain a third matrix; and the output time-space conversion module is used for performing complex weighted channel fusion operation on the third matrix to obtain an output one-dimensional matrix. According to the invention, the use of photoelectric conversion and chip separation equipment is avoided, an all-optical nonlinear convolution system is realized, and a large deep all-optical network with high integration level, high speed and strong expansibility can be formed by cascading.

Description

Technical Field

[0001] The present disclosure relates to the field of optical computing technology, and in particular to an intelligent optical computing nonlinear convolution chip system, method and architecture. Background Art

[0002] With the rapid development of artificial intelligence and scientific computing, the complexity and scale of computing needs are also increasing. However, existing electronic computing hardware (such as CPU, GPU, FPGA, ASIC) will find it difficult to bear the significant computing burden and face the saturation of computing speed and energy efficiency. In the context of the post-Moore era and the interweaving of general artificial intelligence (AGI), realizing the high computing potential of computing architecture has become a persistent goal of high-performance computing research. Light has the advantages of high throughput and low latency during propagation. Based on this, photonic computing units can continuously process input data at high throughput as it flows through the processor, making vowel recognition, serial data classification and time series prediction possible. Summary of the invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first purpose of the present disclosure is to propose an intelligent optical computing nonlinear convolution chip system, which realizes nonlinear activation and information delay through the integrated first microring resonator array and the second microring resonator array, and converts the input one-dimensional matrix into matrix-vector multiplication through the interleaved time and space dimensions through the first phase modulator array and the second phase modulator array, avoiding the use of photoelectric conversion and off-chip devices, and realizing an all-optical nonlinear convolution system, which can be directly cascaded to form a large-scale deep all-optical network, has the advantages of high integration, high speed, and strong scalability, and accelerates the development of machine vision to the sub-nanosecond level. At the same time, the intelligent optical computing nonlinear convolution chip system constructs a complex temporal convolutional neural network, which has superior performance in image classification, high-speed dynamic video classification, and human motion generation tasks.

[0005] The second objective of the present disclosure is to provide an image classification method.

[0006] To achieve the above-mentioned purpose, the first embodiment of the present disclosure proposes an intelligent optical computing nonlinear convolution chip system, the system comprising an input spatiotemporal conversion module, an input temporal coding and nonlinear activation module, a one-dimensional convolution processing module, an output temporal coding and nonlinear activation module, and an output spatiotemporal conversion module, wherein: The input space-time conversion module is used to obtain an input one-dimensional matrix to be analyzed, and to process the input one-dimensional matrix through a first phase modulator array to generate a complex weighted extended channel matrix corresponding to the input one-dimensional matrix; The input timing coding and nonlinear activation module is used to perform nonlinear activation and first information delay on the complex weighted extended channel matrix through the first microring resonator array to obtain a corresponding first matrix; The one-dimensional convolution processing module is used to perform matrix multiplication on the first matrix through a second phase modulator array to obtain a corresponding second matrix; The output timing coding and nonlinear activation module is used to perform nonlinear activation on the second matrix through a second microring resonator array and delay the second information to obtain a corresponding third matrix; The output space-time conversion module is used to perform a complex-weighted channel fusion operation on the third matrix through the first phase modulator array to obtain an output one-dimensional matrix.

[0007] Optionally, the first phase modulator array is implemented by a Mach-Zehnder interferometer to determine the amplitude modulation coefficient and phase modulation coefficient of each channel for light.

[0008] Optionally, the first microring resonator array completes information delay through a group delay effect, and realizes all-optical nonlinearity through Kerr nonlinearity and free carrier dispersion effect.

[0009] Optionally, the intelligent optical computing nonlinear convolution chip system is used for image classification, high-speed dynamic video classification and human motion generation tasks.

[0010] To achieve the above-mentioned purpose, the second aspect of the present disclosure proposes an image classification method, including: Acquire the image to be analyzed; Performing data preprocessing on the image to obtain a corresponding one-dimensional image matrix; Performing feature extraction on the one-dimensional image matrix through a target feature extraction model to obtain a corresponding target feature matrix, wherein the target feature extraction model is composed of at least one intelligent optical computing nonlinear convolution chip system; Based on the target feature matrix, a classification result of the image is determined.

[0011] Optionally, before extracting features from the one-dimensional image matrix through a target feature extraction model to obtain a corresponding target feature matrix, the method further includes: Determine the number of layers of the convolutional architecture and the number of image categories; Based on the number of layers of the convolutional architecture, the intelligent optical computing nonlinear convolutional chip system is connected in series to obtain each layer of the convolutional network; Based on the number of image classifications, each layer of the convolutional network is connected in parallel to obtain an initial feature extraction model.

[0012] Optionally, extracting features from the one-dimensional image matrix through a target feature extraction model to obtain a corresponding target feature matrix includes: Inputting the one-dimensional image matrix into each layer of the convolution network respectively, and sequentially passing through the intelligent optical computing nonlinear convolution chip system connected in series in each layer of the convolution network, to obtain the one-dimensional feature matrix of each layer of the convolution network; The one-dimensional feature matrices are concatenated to obtain corresponding target feature matrices.

[0013] Optionally, determining the classification result of the image based on the target feature matrix includes: determining the classification result of the image by detecting and accumulating output light intensity through a photodetector based on the target feature matrix.

[0014] Another object of the present invention is to provide an image classification device, characterized in that it includes: An acquisition module, used for acquiring images to be analyzed; A data processing module, used for performing data preprocessing on the image to obtain a corresponding one-dimensional image matrix; A feature extraction module, used to extract features from the one-dimensional image matrix through a target feature extraction model to obtain a corresponding target feature matrix, wherein the target feature extraction model is composed of at least one intelligent optical computing nonlinear convolution chip system; A classification module is used to determine a classification result of the image based on the target feature matrix.

[0015] In summary, the intelligent optical computing nonlinear convolution chip system, method and architecture provided by the present disclosure realize nonlinear activation and information delay through the integrated first microring resonator array and the second microring resonator array, and convert the input one-dimensional matrix into matrix-vector multiplication through the interleaved time and space dimensions through the first phase modulator array and the second phase modulator array, avoiding the use of photoelectric conversion and off-chip devices, realizing an all-optical nonlinear convolution system, which can be directly cascaded to form a large-scale deep all-optical network, has the advantages of high integration, high speed and strong scalability, and accelerates the development of machine vision to the sub-nanosecond level. At the same time, the intelligent optical computing nonlinear convolution chip system constructs a complex temporal convolutional neural network, which has superior performance in image classification, high-speed dynamic video classification and human motion generation tasks.

[0016] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1 A schematic diagram of the structure of an intelligent optical computing nonlinear convolution chip system provided by an embodiment of the present disclosure; Figure 2 A schematic diagram of the optical structure of an intelligent optical computing nonlinear convolution chip system provided by an embodiment of the present disclosure; Figure 3 A schematic diagram of simulation and experimental output and depth detection results of a nonlinear convolution chip system based on intelligent optical computing provided in an embodiment of the present disclosure; Figure 4 A network structure diagram generated by human actions provided by an embodiment of the present disclosure; Figure 5 A schematic diagram of the architecture of an intelligent optical computing nonlinear convolution chip system according to an embodiment of the present disclosure; Figure 6 A schematic diagram of a flow chart of an image classification method proposed in an embodiment of the present disclosure; Figure 7 A schematic diagram of an image classification method proposed in an embodiment of the present disclosure; Figure 8 A schematic diagram of the structure of an image classification device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0019] Currently, photonic processing units still face challenges in achieving integrated and scalable nonlinear activation, limiting the potential of large-scale photonic neural networks and complex artificial intelligence tasks. For example, optical nonlinearity can be achieved through laser cooling of atoms, photorefractive crystals, and light scattering, but the above free-space solutions are difficult to integrate on-chip, which weakens their compatibility, scalability, and stability. On the other hand, on-chip nonlinear activation mainly relies on photoelectric conversion and transimpedance amplifiers, which require heterogeneous integration with CMOS electronic devices, resulting in low energy efficiency, redundant delays, and increased system complexity. Based on this, all-optical integrated nonlinear activation is crucial for the next generation of photonic neural networks.

[0020] The present disclosure is described in detail below with reference to specific embodiments.

[0021] Figure 1 This is a schematic diagram of the structure of an intelligent optical computing nonlinear convolution chip system provided by an embodiment of the present disclosure. Figure 1As shown, the model includes an input space-time conversion module 101, an input time series coding and non-linear activation module 102, a one-dimensional convolution processing module 103, an output time series coding and non-linear activation module 104 and an output space-time conversion module 105, wherein: An input space-time conversion module 101 is used to obtain an input one-dimensional matrix to be analyzed, and to process the input one-dimensional matrix through a first phase modulator array to generate a complex weighted extended channel matrix corresponding to the input one-dimensional matrix; An input timing coding and nonlinear activation module 102 is used to perform nonlinear activation and first information delay on the complex weighted extended channel matrix through a first microring resonator array to obtain a corresponding first matrix; A one-dimensional convolution processing module 103, configured to perform matrix multiplication on the first matrix through a second phase modulator array to obtain a corresponding second matrix; The output timing coding and nonlinear activation module 104 is used to perform nonlinear activation on the second matrix and delay the second information through the second microring resonator array to obtain a corresponding third matrix; The output space-time conversion module 105 is used to perform a complex-weighted channel fusion operation on the third matrix through the first phase modulator array to obtain an output one-dimensional matrix.

[0022] In one embodiment of the present disclosure, the above-mentioned intelligent optical computing nonlinear convolution chip system can be applied to a variety of tasks, including but not limited to image classification, high-speed dynamic video classification and human motion generation tasks.

[0023] Among them, in one embodiment of the present disclosure, the above-mentioned input one-dimensional matrix to be analyzed can be an input one-dimensional matrix obtained after data preprocessing of input data of different tasks. For example, in one embodiment of the present disclosure, assuming that the above-mentioned intelligent optical computing nonlinear convolution chip system is suitable for image classification tasks, the input one-dimensional matrix obtained by input is an input one-dimensional matrix obtained by data processing of the image.

[0024] And, in one embodiment of the present disclosure, the first phase modulator array can be implemented by a Mach-Zehnder interferometer to determine the amplitude modulation coefficient and phase modulation coefficient of each channel for light. And, in one embodiment of the disclosure, the input space-time conversion module 101 can weight the channel information of the input one-dimensional matrix through the first phase modulator array, and increase or decrease the number of channels to achieve the change of the spatial dimension of the information.

[0025] In one embodiment of the present disclosure, the first microring resonator array may be a tunable microring resonator. In one embodiment of the present disclosure, the first microring resonator array completes information delay through group delay effect, and realizes all-optical nonlinearity through Kerr nonlinearity and free carrier dispersion effect. And, in one embodiment of the present disclosure, the first microring resonator array may change the reflectivity of the microring through a thermoelectric modulator, thereby changing the free carrier dispersion effect and group delay effect of the microring.

[0026] And, in one embodiment of the present disclosure, the number of microring activations of the first microring resonator array and the second microring resonator array may be different. Specifically, in one embodiment of the present disclosure, the number of microring activations corresponding to the first microring resonator array may be 12, 8, 4, 0; and the number of microring activations corresponding to the second microring resonator array may be 3, 2, 1, 0.

[0027] Furthermore, in one embodiment of the present disclosure, the all-optical nonlinearity achieved by the first microring resonator array avoids the use of photoelectric conversion and off-chip devices, thereby enabling the intelligent optical computing nonlinear convolution chip system to constitute an intelligent optical computing nonlinear convolution chip.

[0028] Figure 2 A schematic diagram of the optical structure of an intelligent optical computing nonlinear convolution chip system provided in an embodiment of the present disclosure.

[0029] Figure 3 The schematic diagram of simulation and experimental output and depth detection results of the nonlinear convolution chip system based on intelligent optical computing provided by the embodiment of the present disclosure. Figure 3 a is an optical micrograph of the entire chip; Figure 3 b is an optical micrograph of a microring resonator (MRR) and a Mach-Zehnder interferometer (MZI); Figure 3 c is the optical time delay using the pulse signal as input when activating 2 MRRs or 4 MRRs (corresponding to the left and right panels, respectively), and the shaded area indicates the delay time; Figure 3 d is the time delay when activating different numbers of MRRs; Figure 3 e : Experimental intensity measurements and the relationship between phase change and input intensity when 2 or 4 MRRs are activated (left and middle panels); Figure 3 f Phase nonlinear performance under different numbers of activated MRRs.

[0030] In one embodiment of the present disclosure, the above-mentioned intelligent optical computing nonlinear convolution chip system establishes an all-optical computing system with an all-optical nonlinear structure and high-order convolution, and shows excellent performance compared to the electronic network with the same structure (taking the full convolution structure network with ReLU nonlinearity as an example). Among them, in one embodiment of the present disclosure, the above-mentioned intelligent optical computing nonlinear convolution chip system can work normally under long distance (up to 6 meters), low power (minimum 0.3µW), high speed (up to 600Hz), and high transmittance conditions, proving the performance advantage of the intelligent optical computing nonlinear convolution chip system over traditional depth perception methods such as LiDAR. The reconfigurable neuron activation composed of the intelligent optical computing nonlinear convolution chip system makes the delay as short as 0.257 nanoseconds, and achieves a computing power of 4.96 TOPS on a single chip.

[0031] And, in one embodiment of the present disclosure, the TPPU network composed of the above-mentioned intelligent optical computing nonlinear convolution chip system was experimentally evaluated on the image classification task, achieving an accuracy of 90.1% (84.2% without nonlinearity). Among them, in one embodiment of the present disclosure, the extended TPPU network demonstrated the ability to perform more complex human motion generation, and its FID (Fréchet distance) fidelity index was 8.424 (14.77 without nonlinearity).

[0032] Furthermore, in one embodiment of the present disclosure, the intelligent optical computing nonlinear convolution chip system can be directly cascaded to form a large-scale optical network. In a deep network, an optical fiber amplifier can be used to strengthen the connection and enhance the light intensity, so that the deep network can work normally.

[0033] For example, Figure 4 A network structure diagram generated by human actions provided by an embodiment of the present disclosure, such as Figure 4 As shown in the figure, the human action generation network builds an encoder-decoder architecture by forming a basic general stack of multiple intelligent optical computing nonlinear convolution chip systems. This encoder-decoder architecture has the task of complex mapping between input and output sequences and can extract features between elements in the sequence. Figure 4 As shown in the figure, each frame of the human action sequence contains the three-dimensional Cartesian coordinates of 20 key joints of the human body. During the generation process, the joint position information of the four frames before and after the current frame is used as input. In addition, during the training process of the above human action generation network, the Markov chain Monte Carlo method is used to optimize the generated action sequence, and the Fleche distance (FID) and kernel distance (KID) are used to evaluate the feature extraction quality of the generated action. Among them, the smaller the value, the higher the similarity between the features of the generated action and the features of the original action.

[0034] In one embodiment of the present disclosure, during the training of the human action generation network, the corresponding loss function can be defined as:

[0035] in Indicates the first The true value of the spatial Cartesian coordinates of the nodes, Represents the predicted network coordinate value, n is the total number of joints, which can be set as needed. For example, n=20.

[0036] Figure 5 This is a schematic diagram of the architecture of the intelligent optical computing nonlinear convolution chip system provided by the embodiment of the present disclosure. Figure 5 As shown, the architecture uses a thermoelectric cooler (TED200C, Thorlabs) with a slow proportional-integral-derivative feedback loop to stabilize the temperature of the chip, and the electrical probe contacts the micro-heating element. In order to simultaneously control the states of the micro-ring resonator (MRR) and the phase regulator that need to be modulated, the architecture uses 12 6-channel isolated arbitrary waveform generators (PXI-7961, CHNNI Instruments) as the input of the device's photothermal controller. The beam is generated by a tunable solid-state laser (CoBriteDX–Tunable Laser) with an operating wavelength of 1550 nm. The beam is coupled to a lithium niobate electro-optic modulator (LN81S-FC, Thorlabs) through a single-mode fiber to modulate the intensity of the beam. The arbitrary waveform generator (AWG70001B, Tektronix) converts the data into an electrical waveform of a specific frequency. The optical signal is polarized by a linear polarizer (MPC320, Thorlabs) and coupled to the input channel of the chip as the system input signal. The output channel of the chip was connected to an InGaAs photodetector (DET08CFC / M, Thorlabs), which converted the output light intensity into an electrical signal and captured it by a high-speed oscilloscope (MSO64B, Tektronix).

[0037] The intelligent optical computing nonlinear convolution chip system of the disclosed embodiment realizes nonlinear activation and information delay through the integrated first microring resonator array and the second microring resonator array, and converts the input one-dimensional matrix into matrix-vector multiplication through the interleaved time and space dimensions through the first phase modulator array and the second phase modulator array, avoiding the use of photoelectric conversion and off-chip devices, realizing an all-optical nonlinear convolution system, which can be directly cascaded to form a large-scale deep all-optical network, has the advantages of high integration, high speed, and strong scalability, and accelerates the development of machine vision to the sub-nanosecond level. At the same time, the intelligent optical computing nonlinear convolution chip system constructs a complex temporal convolutional neural network, which has superior performance in image classification, high-speed dynamic video classification, and human motion generation tasks.

[0038] In order to implement the above embodiment, Figure 6 The present disclosure also proposes an image classification method, which may include the following steps: Step 601, obtaining an image to be analyzed; Step 602, preprocessing the image to obtain a corresponding one-dimensional image matrix; Step 603, extracting features from the one-dimensional image matrix through a target feature extraction model to obtain a corresponding target feature matrix; Step 604: Determine the classification result of the image based on the target feature matrix.

[0039] In one embodiment of the present disclosure, after acquiring the image to be analyzed, the image may be preprocessed to obtain a corresponding one-dimensional image matrix. Specifically, in one embodiment of the present disclosure, the multi-dimensional matrix corresponding to the image may be rearranged to obtain a corresponding one-dimensional image matrix.

[0040] In one embodiment of the present disclosure, the above-mentioned target feature extraction model is composed of at least one intelligent optical computing nonlinear convolution chip system.

[0041] In one embodiment of the present disclosure, before extracting features from a one-dimensional image matrix through a target feature extraction model to obtain a corresponding target feature matrix, the method may further include the following steps: Step 1: Determine the number of layers of the convolutional architecture and the number of image categories; Step 2: Based on the number of layers of the convolutional architecture, the intelligent optical computing nonlinear convolutional chip system is connected in series to obtain each layer of the convolutional network; Step 3: Based on the number of image classifications, each layer of the convolutional network is connected in parallel to obtain the initial feature extraction model.

[0042] In one embodiment of the present disclosure, the number of layers of the convolutional architecture and the number of image classifications can be determined by user input or manual experience. In one embodiment of the present disclosure, the target feature extraction model can be composed of an N-layer fully convolutional architecture, and for M classification tasks, the network has M channels. Each layer contains N×M×4 convolution kernels. For example, in one embodiment of the present disclosure, the number of layers of the above convolutional architecture is 5, and the number of image classifications is 4.

[0043] Among them, in one embodiment of the present disclosure, after the initial feature extraction model is obtained through the above steps, the initial feature extraction model can be trained, and the trained feature extraction model is determined as the target feature extraction model. In one embodiment of the present disclosure, the method for training the initial feature extraction model is the same as the existing training method, and the present embodiment is not repeated here.

[0044] In one embodiment of the present disclosure, the loss function in the above training process may be: , , , in, represents the cross entropy loss function, represents the energy distribution loss function, represents the number of categories, is the indicator function (0 or 1), when the sample The true category is The value is 1 when , otherwise it is 0; prediction probability is the observed value Belongs to category The predicted probability of represents the predicted probability distribution ratio, is the number of samples.

[0045] In one embodiment of the present disclosure, the cross entropy loss function and the energy distribution loss function in the above loss function can guide the optimization of the output energy distribution to ensure that the optical experimental results are consistent with the simulation results.

[0046] Furthermore, in one embodiment of the present disclosure, the method of extracting features from a one-dimensional image matrix through a target feature extraction model to obtain a corresponding target feature matrix may include the following steps: Step 6031, inputting the one-dimensional image matrix into each layer of the convolutional network respectively, and sequentially passing through the intelligent optical computing nonlinear convolution chip system connected in series in each layer of the convolutional network to obtain a one-dimensional feature matrix of each layer of the convolutional network; Step 6032, concatenate the one-dimensional feature matrices to obtain the corresponding target feature matrix.

[0047] Furthermore, in one embodiment of the present disclosure, after the target feature matrix is ​​obtained through the above steps, the classification result of the image can be determined based on the target feature matrix.

[0048] Specifically, in one embodiment of the present disclosure, a method for determining a classification result of an image based on a target feature matrix may include: determining the classification result of the image based on the target feature matrix by detecting and accumulating output light intensity through a photodetector.

[0049] Specifically, in one embodiment of the present disclosure, each channel corresponds to a classification category. Within a specified number of time steps, the output light intensity of each channel is accumulated to obtain the accumulated light intensity of each channel. The channel corresponding to the maximum value of the accumulated light intensity reflects the category of the image.

[0050] Based on the above description, Figure 7 FIG. 1 is a schematic diagram of an image classification method proposed in an embodiment of the present disclosure. Figure 7 As shown, The input image is processed by data preprocessing, and the corresponding original data 4×4 is rearranged to obtain an input one-dimensional image matrix of 1×16. Then, each layer of the 4-layer convolutional network in the target feature extraction model is used to obtain a one-dimensional feature matrix of 1×16 corresponding to each layer of the convolutional network. The one-dimensional feature matrices of the 4-layer convolutional network are spliced ​​to obtain a 4×1×16 target feature matrix. Based on the target feature matrix, the classification result of the image is determined by detecting and accumulating the output light intensity through a photoelectric detector. Among them, the feature extraction model includes connecting two intelligent optical computing nonlinear convolutional chip systems in series to obtain each layer of the convolutional network, and includes 4 layers of convolutional networks in parallel.

[0051] In the disclosed embodiment, the above-mentioned image classification method has strong nonlinear performance and characterization ability through the target feature extraction model composed of the intelligent optical computing nonlinear convolution chip system, thereby making the image classification result more accurate and improving the image classification accuracy.

[0052] In order to implement the above embodiment, Figure 8 The present disclosure also proposes an image classification device, which may include: An acquisition module 801 is used to acquire an image to be analyzed; The data processing module 802 is used to perform data preprocessing on the image to obtain a corresponding one-dimensional image matrix; A feature extraction module 803 is used to extract features from the one-dimensional image matrix through a target feature extraction model to obtain a corresponding target feature matrix, wherein the target feature extraction model is composed of at least one intelligent optical computing nonlinear convolution chip system; The classification module 804 is used to determine the classification result of the image based on the target feature matrix.

[0053] In one embodiment of the present disclosure, an array of intelligent optical computing nonlinear convolution chip systems is divided into different groups, and different groups of intelligent optical computing nonlinear convolution chip systems correspond to different wavelengths in the preset incident light.

[0054] In one embodiment of the present disclosure, the above device is also used for: Determine the number of layers of the convolutional architecture and the number of image categories; Based on the number of layers of the convolutional architecture, the intelligent optical computing nonlinear convolution chip system is connected in series to obtain each layer of the convolutional network; Based on the number of image classifications, each layer of the convolutional network is connected in parallel to obtain the initial feature extraction model.

[0055] In one embodiment of the present disclosure, the feature extraction module 803 is specifically used to: The one-dimensional image matrix is ​​input into each layer of the convolutional network respectively, and the one-dimensional feature matrix of each layer of the convolutional network is obtained by sequentially passing through the intelligent optical computing nonlinear convolution chip system connected in series in each layer of the convolutional network; The one-dimensional feature matrices are concatenated to obtain the corresponding target feature matrix.

[0056] In one embodiment of the present disclosure, the classification module 804 is specifically used to: Based on the target feature matrix, the image classification result is determined by detecting and accumulating the output light intensity through the photodetector.

[0057] In the disclosed embodiment, the above-mentioned image classification system has strong nonlinear performance and characterization ability through the target feature extraction model composed of the intelligent optical computing nonlinear convolution chip system, thereby making the image classification result more accurate and improving the image classification accuracy.

[0058] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this disclosure shall comply with the relevant laws and regulations and shall not violate public order and good morals.

[0059] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0060] The present disclosure anticipates providing implementation schemes for users to selectively block the use or access of personal information data. That is, the present disclosure anticipates providing hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, risks can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0061] The acquisition, transmission, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations.

[0062] It should be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0063] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they contradict each other.

[0064] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0065] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.

[0066] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0067] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0068] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0069] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0070] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.

Claims

1. An intelligent optical computing nonlinear convolution chip system, characterized in that: The system includes an input space-time conversion module, an input time series coding and non-linear activation module, a one-dimensional convolution processing module, an output time series coding and non-linear activation module, and an output space-time conversion module, wherein: The input space-time conversion module is used to obtain an input one-dimensional matrix to be analyzed, and to process the input one-dimensional matrix through a first phase modulator array to generate a complex weighted extended channel matrix corresponding to the input one-dimensional matrix; The input timing coding and nonlinear activation module is used to perform nonlinear activation and first information delay on the complex weighted extended channel matrix through the first microring resonator array to obtain a corresponding first matrix; The one-dimensional convolution processing module is used to perform matrix multiplication on the first matrix through a second phase modulator array to obtain a corresponding second matrix; The output timing coding and nonlinear activation module is used to perform nonlinear activation on the second matrix through a second microring resonator array and delay the second information to obtain a corresponding third matrix; The output space-time conversion module is used to perform a complex-weighted channel fusion operation on the third matrix through the first phase modulator array to obtain an output one-dimensional matrix.

2. The system according to claim 1, characterized in that The first phase modulator array is implemented by a Mach-Zehnder interferometer to determine the amplitude modulation coefficient and phase modulation coefficient of each channel for light.

3. The system according to claim 1, characterized in that The first microring resonator array completes information delay through the group delay effect, and realizes all-optical nonlinearity through Kerr nonlinearity and free carrier dispersion effect.

4. The intelligent optical computing nonlinear convolution chip system according to any one of claims 1 to 3, characterized in that: Used for image classification, high-speed dynamic video classification, and human motion generation tasks.

5. An image classification method, characterized in that: include: Acquire the image to be analyzed; Performing data preprocessing on the image to obtain a corresponding one-dimensional image matrix; Performing feature extraction on the one-dimensional image matrix through a target feature extraction model to obtain a corresponding target feature matrix, wherein the target feature extraction model is composed of at least one intelligent optical computing nonlinear convolution chip system according to any one of claims 1 to 4; Based on the target feature matrix, a classification result of the image is determined.

6. The method according to claim 5, characterized in that Before extracting features from the one-dimensional image matrix through a target feature extraction model to obtain a corresponding target feature matrix, the method further includes: Determine the number of layers of the convolutional architecture and the number of image categories; Based on the number of layers of the convolutional architecture, the intelligent optical computing nonlinear convolutional chip system is connected in series to obtain each layer of the convolutional network; Based on the number of image classifications, each layer of the convolutional network is connected in parallel to obtain an initial feature extraction model.

7. The method according to claim 6, characterized in that The step of extracting features from the one-dimensional image matrix through a target feature extraction model to obtain a corresponding target feature matrix includes: Inputting the one-dimensional image matrix into each layer of the convolution network respectively, and sequentially passing through the intelligent optical computing nonlinear convolution chip system connected in series in each layer of the convolution network, to obtain the one-dimensional feature matrix of each layer of the convolution network; The one-dimensional feature matrices are concatenated to obtain corresponding target feature matrices.

8. The method according to claim 5, characterized in that Determining the classification result of the image based on the target feature matrix includes: determining the classification result of the image by detecting and accumulating output light intensity through a photoelectric detector based on the target feature matrix.

9. An image classification device, characterized in that: include: An acquisition module, used for acquiring images to be analyzed; A data processing module, used for performing data preprocessing on the image to obtain a corresponding one-dimensional image matrix; A feature extraction module, used for performing feature extraction on the one-dimensional image matrix through a target feature extraction model to obtain a corresponding target feature matrix, wherein the target feature extraction model is composed of at least one intelligent optical computing nonlinear convolution chip system according to any one of claims 1 to 4; A classification module is used to determine a classification result of the image based on the target feature matrix.

10. The device according to claim 9, characterized in that The device is also used for: Determine the number of layers of the convolutional architecture and the number of image categories; Based on the number of layers of the convolutional architecture, the intelligent optical computing nonlinear convolutional chip system is connected in series to obtain each layer of the convolutional network; Based on the number of image classifications, each layer of the convolutional network is connected in parallel to obtain an initial feature extraction model.

Images