Optical neural network modulation-activation-propagation model and architecture

Abstract

The invention relates to the technical field of optical computing, in particular to an optical neural network modulation-activation-propagation model and architecture. The model comprises a spatial modulation module used for performing spatial modulation on input scene information to obtain an optical signal after spatial modulation; the neuron activation module is used for converting the spatially modulated optical signals into transmission optical signals, activating neuron subsets corresponding to the transmission optical signals in a neuron set and converting the transmission optical signals into multispectral signals through the activated neuron subsets, and neurons in the neuron set interact in an all-optical connection mode; and the signal propagation module is used for outputting a multispectral signal. By adopting the scheme, the end-to-end all-optical intelligent sensing calculation from light sensing to light processing can be realized.

Description

Technical Field

[0001] The present disclosure relates to the field of optical computing technology, and in particular to an optical neural network modulation-activation-propagation model and architecture. Background Art

[0002] With the rapid development of artificial intelligence and scientific computing, the complexity and scale of computing needs are also increasing. However, the existing electronic computing technology is limited by Moore's Law, and its performance is gradually approaching saturation, making it difficult to effectively cope with the increasingly stringent requirements of large-scale complex algorithms for computing power and power consumption. Light has natural advantages such as high throughput and low latency in the propagation process. Optical computing technology that uses photons instead of electrons as computing carriers is seen as the key to breaking the existing computing bottleneck.

[0003] However, existing optical computing techniques cannot directly process spatial intensity images. Most optical image processors rely on electronic cameras, which requires recording spatial images first and then feeding them into the optical computing module, or collecting pre-processing results and passing them to electronic post-processing systems. Summary of the invention

[0004] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the first objective of the present disclosure is to propose an optical neural network modulation-activation-propagation model to achieve end-to-end all-optical intelligent sensing and computing from light perception to light processing.

[0006] The second objective of the present disclosure is to propose an optical neural network modulation-activation-propagation architecture.

[0007] To achieve the above objectives, the first embodiment of the present disclosure proposes an optical neural network modulation-activation-propagation model, including: A spatial modulation module, used to perform spatial modulation on the input scene information to obtain a spatially modulated optical signal; A neuron activation module, used for converting the spatially modulated optical signal into a transmission optical signal, and activating a subset of neurons in a neuron set corresponding to the transmission optical signal, and converting the transmission optical signal into a multi-spectral signal through the activated subset of neurons, wherein the neurons in the neuron set interact with each other in an all-optical connection manner; A signal propagation module is used to output the multi-spectral signal.

[0008] Optionally, the spatial modulation module is used to perform spatial modulation on the input scene information to obtain a spatially modulated optical signal, specifically for: Optical elements are used to dynamically control the phase, amplitude and polarization state of the input scene information to obtain a spatially modulated optical signal that meets the control requirements.

[0009] Optionally, the optical element includes at least one of the following: Spatial light modulator; Polarizer; Beam splitter.

[0010] Optionally, the regulation requirement is determined based on a simulation network after training and optimization, and the simulation network is a network built through physical modeling that is consistent with the spatial modulation module.

[0011] Optionally, when the neuron activation module is used to activate a subset of neurons corresponding to the transmission light signal in a neuron set, it is specifically used to: Determining the response relationship between light signals of different wavelengths in the transmitted light signal and neurons when responding; A subset of neurons corresponding to the transmitted light signal in the neuron set is activated according to the response relationship.

[0012] Optionally, the neuron activation module uses a sensing chip, the sensing chip is composed of a plurality of sensing units, the sensing unit is composed of a resonant ring, and the neuron activation module is used to activate a subset of neurons corresponding to the transmission light signal in a neuron set, specifically for: The resonant ring corresponding to the transmission optical signal in the sensing chip is activated.

[0013] Optionally, when the neuron activation module is used to convert the transmission light signal into a multi-spectral signal through the activated neuron subset, it is specifically used to: The transmission light signal is input into the activated resonant ring, so that the activated resonant ring changes the resonant wavelength and the resonant depth through the carrier drift effect, thereby obtaining a multi-spectral signal.

[0014] Optionally, when the signal propagation module is used to output the multi-spectral signal, it is specifically used to: The multi-spectral signal is transmitted to an information receiving end via an optical fiber.

[0015] Optionally, the information receiving end includes at least one of the following: Passive receiving terminal; An intelligent system capable of deep learning and reasoning on the multispectral signals.

[0016] To achieve the above-mentioned purpose, the second aspect of the present disclosure proposes an optical neural network modulation-activation-propagation architecture, including: the optical neural network modulation-activation-propagation model shown in any one of the aforementioned first aspects.

[0017] In summary, the optical neural network modulation-activation-propagation model and architecture provided in the present invention can realize end-to-end all-optical intelligent sensing and computing from light perception to light processing by achieving fast and efficient information processing through the coordinated work of various stages among modulation, activation and propagation.

[0018] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1 A schematic diagram of the structure of an optical neural network modulation-activation-propagation model provided by an embodiment of the present disclosure; Figure 2 A schematic diagram of the structure of an optical neural network modulation-activation-propagation architecture provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0021] With the widespread deployment of sensors and computing modules at the edge of the Internet, the importance of image processing, transmission, and reconstruction has become increasingly prominent. Optical technology plays a vital role in the fields of imaging, communication, and display, while digital complementary metal oxide semiconductor (CMOS) electronics dominates image sensing and processing. The independent development of optics and electronics has successfully improved their respective transmission bandwidths and processing speeds. However, the frame rate and energy efficiency of operations such as spatial image processing, transmission, and reconstruction have approached their physical limits. On the one hand, the gradual saturation of electronic circuit performance has significantly limited the further improvement of signal processing speed and energy efficiency; on the other hand, during image sensing and transmission, operations require frequent serial conversions between the electronic domain and the optical domain, which not only increases processing time, but also generates a lot of power, thereby limiting the frame rate of operations.

[0022] In order to effectively alleviate the bottleneck problem of integrated electronic circuits, performing image processing in parallel directly in the optical domain is considered to be a potential solution. At present, some researchers have begun to use spatial optical modulation elements to pre-process images and input the results into electronic systems for post-processing. However, existing photon computing methods are still unable to directly process spatial intensity images. Most optical image processors rely on electronic cameras, which requires recording spatial images first and then inputting them into optical computing modules, or collecting pre-processing results and passing them to electronic post-processing systems.

[0023] The present disclosure is described in detail below with reference to specific embodiments.

[0024] Figure 1 This is a schematic diagram of the structure of an optical neural network modulation-activation-propagation model provided by an embodiment of the present disclosure. Figure 1 As shown, the optical neural network modulation-activation-propagation model includes: Spatial modulation module, used to input scene information ( x 1,1 … x M,N ) is spatially modulated to obtain the spatially modulated optical signal ( h 1,1 … h M,N ); Neuron activation module, used to convert the spatially modulated optical signal ( h 1,1 … h M,N ) is converted into a transmission light signal, and a subset of neurons corresponding to the transmission light signal in the neuron set is activated. The transmission light signal is converted into a multi-spectral signal ( λ 1…… λ k ), in which neurons in a neuron ensemble interact with each other via all-optical connections; Signal propagation module, used to output multi-spectral signals ( λ 1…… λ k ).

[0025] Wherein, M and N are the length and width of the original image corresponding to the input scene information; k The dimension of the output data is determined by the task to be completed (the number of categories or the hidden layer dimension).

[0026] In summary, the model provided in this embodiment can realize fast and efficient information processing by cooperating between the modulation, activation and propagation stages, and can realize an end-to-end solution. By driving light sensing through optical calculation, it strives to realize nanosecond-level "light speed" intelligent sensing calculation in natural scenes, which can greatly improve the efficiency and performance of image processing and is expected to provide new ideas and directions for future image processing systems. Optionally, the spatial modulation module is used to perform spatial modulation on the input scene information to obtain a spatially modulated optical signal, specifically for: Optical elements are used to dynamically control the phase, amplitude and polarization state of the input scene information to obtain a spatially modulated optical signal that meets the control requirements.

[0027] According to some embodiments, the input scene information may be in the form of a light wave, for example. By dynamically regulating the phase, amplitude and polarization state of the light wave, a spatially modulated light signal that meets the regulation requirements is obtained, which can ensure that the information is transmitted to subsequent components in the best form.

[0028] In some embodiments, the control requirements can be determined based on the trained and optimized simulation network, which is a network built through physical modeling and consistent with the spatial modulation module. Therefore, by training and optimizing the simulation network, all weights in the simulation network converge to the optimal state, and using the network parameters at this time as the parameters used in the control requirements, it can be ensured that the information of the optical signal after spatial modulation reaches the optimal form.

[0029] According to some embodiments, the optical element includes, but is not limited to, at least one of the following: Spatial light modulator; Polarizer; Beam splitter.

[0030] In some embodiments, a spatial light modulator is a device that modulates the spatial distribution of light waves. The device is composed of a liquid crystal array, and the phase of the incident light beam is modulated by the birefringence of the liquid crystal molecules. By changing the voltage applied to the liquid crystal pixel molecules, there will be different angles between the liquid crystal molecules and the electric field, that is, the director of the liquid crystal molecules and the polarization direction of the incident light form a certain angle, thereby changing the effective refractive index of the liquid crystal, thereby changing the size of the optical path of the light at that location, and achieving the purpose of phase modulation. By combining a spatial light modulator with other optical devices such as polarizers and beam splitters, the interference characteristics of coherent light are used to change the voltage at both ends of the spatial light modulator in real time, and ultimately the dynamic regulation of the phase, amplitude and polarization state of the light wave can be completed.

[0031] It should be noted that by using optical elements to dynamically control the phase, amplitude and polarization state of the input scene information, effective processing of the input information can be achieved and the effect of spatial modulation can be improved.

[0032] Optionally, when the neuron activation module is used to activate a subset of neurons corresponding to the transmitted light signal in the neuron set, it is specifically used to: Determine the response relationship between light signals of different wavelengths in the transmitted light signal and the response of neurons; The subset of neurons in the neuron set corresponding to the transmitted light signal is activated according to the response relationship.

[0033] According to some embodiments, the neuron activation module uses a sensing chip, which is composed of a plurality of sensing units, and the sensing unit is composed of a resonant ring. That is, when the neuron activation module is used to activate a subset of neurons corresponding to the transmitted light signal in a neuron set, it is specifically used to: Activate the resonant ring corresponding to the transmitted optical signal in the sensing chip.

[0034] In some embodiments, the resonant ring has a resonance effect on the intensity and phase of light with a wavelength near the resonant wavelength. Using the resonant ring to respond to transmission optical signals of different wavelengths can improve the response effect.

[0035] It should be noted that the transmission light signal refers to the light signal that can be used for calculation. The spatially modulated light signal activates the resonant ring on the sensor chip, causing its resonance peak to shift to different degrees, thereby generating different transmission curves and obtaining the transmission light signal. When the transmission light signal passes through this activated neuron, the information conversion and loading will be realized due to the change in the transmission curve.

[0036] According to some embodiments, when the neuron activation module is used to convert the transmission light signal into a multi-spectral signal through the activated neuron subset, it is specifically used to: The transmission light signal is input into the activated resonant ring, so that the activated resonant ring changes the resonant wavelength and the resonant depth through the carrier drift effect, thereby obtaining a multi-spectral signal.

[0037] In some embodiments, after the transmission light signal is input into the activated resonant ring, it will affect the above-mentioned response relationship through the carrier drift effect, and change the resonance wavelength and resonance depth. For a fixed wavelength, this is equivalent to modulating the intensity and phase of the light at that wavelength, thereby changing the intensity, phase and other information of the transmission light signal. This change contains the information of the modulated light signal.

[0038] In some embodiments, when the intensity and phase of light at a certain wavelength are modulated, lasers with the same intensity and wavelengths near the resonant wavelength are input simultaneously. Through the above-mentioned modulation effect, the transmitted optical signal can be converted into optical signals at different wavelengths, that is, multi-spectral signals.

[0039] According to some embodiments, multiple sensing units in the sensing chip are connected via waveguides, and multi-spectral signals output by different sensing units are coherently superimposed, ultimately resulting in the output of a multi-spectral signal that integrates all sensing units.

[0040] It should be noted that neurons interact with each other through all-optical connections to form a highly interconnected network structure. The activated neurons convert the information they carry into optical signals of different wavelengths through optical modulation technology. This process not only effectively improves the efficiency of information transmission, but also provides multi-dimensional information input for subsequent data processing. Finally, the calculation results are presented in a multi-spectral form. This multi-spectral output form can carry rich information content and facilitate in-depth data analysis.

[0041] Optionally, when the signal propagation module is used to output a multi-spectral signal, it is specifically used to: The multi-spectral signal is transmitted to the information receiving end through optical fiber. This integrated design enables the optical neural network to demonstrate superior performance when processing complex information.

[0042] According to some embodiments, the information receiving end includes but is not limited to at least one of the following: Passive receiving terminal; Intelligent systems capable of deep learning and reasoning on multispectral signals.

[0043] In some embodiments, an advanced intelligent computing algorithm may be integrated into the intelligent system, through which deep learning and reasoning of multi-spectral signals are performed.

[0044] In summary, the model provided in this embodiment has high speed and high parallel processing capabilities, and can meet the requirements of modern intelligent applications for rapid response and efficient processing. Due to the extremely fast propagation speed of optical signals, compared with traditional electronic processing methods, this model can complete the acquisition, processing and analysis of information in a very short time, and is particularly suitable for scenarios that require rapid decision-making such as real-time monitoring and autonomous driving. In addition, the "modulation-activation-propagation" model also has excellent energy consumption performance. Through the effective use of optical components, the model can significantly reduce energy consumption when performing complex computing tasks, which is in line with the current trend of sustainable development. This feature makes optical neural networks more advantageous in large-scale applications and suitable for various high-performance computing environments. In addition, this model can be well compatible with existing optical computing chip methods, breaking through the speed and power consumption constraints brought by optoelectronic / electro-optical conversion, and realizing end-to-end all-optical intelligent sensing from optical perception to optical processing. With its high-speed and high-bandwidth sensing and computing characteristics, it is expected to bring disruptive breakthroughs in performance to fields such as autonomous driving, industrial detection, intelligent robots, virtual reality (VR) / augmented reality (AR), and has broad application prospects.

[0045] In order to implement the above embodiments, the present disclosure also proposes an optical neural network modulation-activation-propagation architecture, including: the optical neural network modulation-activation-propagation model provided by the above embodiments.

[0046] According to some embodiments, Figure 2 This is a schematic diagram of the structure of an optical neural network modulation-activation-propagation architecture provided by an embodiment of the present disclosure. Figure 2 As shown in the figure, it demonstrates a specific application of the modulation-activation-propagation architecture of the optical neural network, which is of great significance in the field of intelligent computing. In this architecture, natural scene information first passes through the modulation stage of the optical neural network. This process uses optical technology to efficiently process the input signal to ensure that the information is transmitted in the best form. The modulated information then enters the activation stage, in which the optical neural network further processes the modulated information through an all-optical connection to form a high-dimensional optical signal output. Next, the modulated and activated optical signal enters the depth perception module, which can intelligently perceive the depth information of the scene, and use the combination of deep learning algorithms and optical elements to achieve rapid recognition and analysis of complex scenes. Throughout the process, the transmission and processing of information are completed at the nanosecond level, which significantly improves the real-time and responsiveness of the system.

[0047] In summary, the architecture provided in this embodiment can realize light-speed intelligent computing and perception at the nanosecond level by realizing photon computing neurons and combining the "modulation-activation-propagation" architecture. This innovative method uses the high-speed characteristics of light to provide a new way of information processing, making the perception and analysis of natural scenes faster and more efficient.

[0048] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this disclosure shall comply with the relevant laws and regulations and shall not violate public order and good morals.

[0049] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0050] The present disclosure anticipates providing implementation schemes for users to selectively block the use or access of personal information data. That is, the present disclosure anticipates providing hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, risks can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0051] The acquisition, transmission, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations.

[0052] It should be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0053] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they contradict each other.

[0054] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0055] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.

[0056] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0057] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0058] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0059] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0060] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.

Claims

1. An optical neural network modulation-activation-propagation model, characterized in that: include: A spatial modulation module, used to perform spatial modulation on the input scene information to obtain a spatially modulated optical signal; A neuron activation module, used for converting the spatially modulated optical signal into a transmission optical signal, and activating a subset of neurons in a neuron set corresponding to the transmission optical signal, and converting the transmission optical signal into a multi-spectral signal through the activated subset of neurons, wherein the neurons in the neuron set interact with each other in an all-optical connection manner; A signal propagation module is used to output the multi-spectral signal.

2. The model according to claim 1, characterized in that The spatial modulation module is used to perform spatial modulation on the input scene information to obtain a spatially modulated optical signal, specifically for: Optical elements are used to dynamically control the phase, amplitude and polarization state of the input scene information to obtain a spatially modulated optical signal that meets the control requirements.

3. The model according to claim 2, characterized in that The optical element comprises at least one of the following: Spatial light modulator; Polarizer; Beam splitter.

4. The model according to claim 1, characterized in that The control requirement is determined based on a simulation network after training and optimization, and the simulation network is a network built through physical modeling that is consistent with the spatial modulation module.

5. The model according to claim 1, characterized in that When the neuron activation module is used to activate the neuron subset corresponding to the transmission light signal in the neuron set, it is specifically used to: Determining the response relationship between light signals of different wavelengths in the transmitted light signal and neurons when responding; A subset of neurons corresponding to the transmitted light signal in the neuron set is activated according to the response relationship.

6. The model according to claim 1, characterized in that The neuron activation module uses a sensing chip, which is composed of a plurality of sensing units, and the sensing unit is composed of a resonant ring. When the neuron activation module is used to activate the neuron subset corresponding to the transmission light signal in the neuron set, it is specifically used to: The resonant ring corresponding to the transmission optical signal in the sensing chip is activated.

7. The model according to claim 6, characterized in that When the neuron activation module is used to convert the transmission light signal into a multi-spectral signal through the activated neuron subset, it is specifically used to: The transmission light signal is input into the activated resonant ring, so that the activated resonant ring changes the resonant wavelength and the resonant depth through the carrier drift effect, thereby obtaining a multi-spectral signal.

8. The model according to claim 1, characterized in that When the signal propagation module is used to output the multi-spectral signal, it is specifically used to: The multi-spectral signal is transmitted to an information receiving end via an optical fiber.

9. The model according to claim 8, characterized in that The information receiving end includes at least one of the following: Passive receiving terminal; An intelligent system capable of deep learning and reasoning on the multispectral signals.

10. An optical neural network modulation-activation-propagation architecture, characterized in that: include: An optical neural network modulation-activation-propagation model as described in any one of claims 1 to 9.

Images