Integrated display system based on optical neural networks

Abstract

The application provides an integrated display system based on an optical neural network, and relates to the technical field of display, and comprises a perception layer, an optical neural network calculation layer and a display layer; the perception layer is used for perceiving external environment signals, modulating input light signals according to the external environment signals, and forming standard light input signals; the optical neural network calculation layer is used for receiving the standard light input signals through a preset weight parameter optical neural network, performing matrix multiplication operation in parallel in the optical domain based on the modulated light signals, performing nonlinear activation output of light field signals, and coupling the light field signals to each display unit of the display layer through an optical output structure; the display unit comprises a photoelectric conversion structure and a light-emitting control structure, the photoelectric conversion structure is used for converting the received coupled light signals into driving electric signals and sending the driving electric signals to the light-emitting control structure, so as to control the light-emitting brightness of the display unit. The integrated display system can effectively reduce the energy consumption and delay of the display system and improve the integration.

Description

Technical Field

[0001] This application relates to the field of display technology, and in particular to an integrated display system based on optical neural networks. Background Technology

[0002] Traditional sensing, computing, and display systems are typically separate. Sensors (such as CMOS and photodiodes) first convert physical signals (such as light intensity, pressure, and temperature) into electrical signals; then, these electrical signals are transmitted to electronic computing chips for processing and calculation; finally, the calculation results need to be converted again by the display driver circuit into a signal format suitable for the display panel before they can be displayed on the screen.

[0003] In traditional discrete architectures, signals need to undergo multiple signal conversions (optical-electrical-optical). At the same time, the transmission of electrical signals between independent modules introduces additional overhead. The combination of these two factors leads to significant latency and power consumption in the data transmission and processing stages. This problem is amplified, especially in high-bandwidth, high-data-volume scenarios such as real-time image processing or AI inference. Summary of the Invention

[0004] In view of this, this application proposes an integrated display system based on optical neural networks.

[0005] This application provides an integrated display system based on optical neural networks, comprising: a perception layer, an optical neural network computing layer, and a display layer;

[0006] The perception layer is used to sense external environmental signals and modulate the input light signal according to the external environmental signals to form a standard light input signal for the corresponding optical neural network.

[0007] The optical neural network computing layer is used to receive the standard optical input signal through an optical neural network with preset weight parameters, perform matrix multiplication operations in parallel in the optical domain based on the modulated optical signal, and perform nonlinear activation to output an optical field signal. The optical field signal is coupled to each display unit of the display layer through an optical output structure, wherein the optical output structure is an optical waveguide structure or a microlens group.

[0008] The display unit includes a photoelectric conversion structure and a light-emitting control structure. The photoelectric conversion structure is used to convert the received coupled optical signal into a driving electrical signal and send it to the light-emitting control structure to control the light-emitting brightness of the display unit. The light-emitting control structure is a liquid crystal cell or a light-emitting element.

[0009] In one embodiment, the sensing layer includes: a photosensitive sensing array, a coupling structure, and an optical modulator;

[0010] The coupling structure is used to couple external incident light to the input waveguide corresponding to the photosensitive sensing array;

[0011] The photosensitive sensing array is used to change its refractive index or absorptivity according to external environmental signals in order to modulate the input optical signal transmitted in the input waveguide.

[0012] The optical modulator is used to encode the optical signal modulated by the photosensitive sensor array to form the standard optical input signal.

[0013] In one embodiment, the optical neural network computation layer includes: a weight configuration module, a nonlinear activation module, and an output module;

[0014] The weight configuration module includes a multi-level optical modulation array. The weight configuration module is used to perform matrix multiplication on the standard optical input signal and to control the phase of the optical modulation array through electro-optic effect or thermo-optic effect to realize weight configuration in matrix multiplication. The optical modulation array is one of MZI array, metasurface array and waveguide array.

[0015] The nonlinear activation module is used to perform nonlinear activation on the standard optical input signal after matrix multiplication and output an optical field signal.

[0016] The output module is used to couple the light field signal to each display unit of the display layer through the light output structure.

[0017] In one embodiment, the nonlinear activation module includes a photodetector, a microelectronic nonlinear unit, and an electro-optic modulator;

[0018] The photodetector is used to convert the standard optical input signal after matrix multiplication into an electrical signal.

[0019] The electro-optic modulator is used to receive the electrical signal processed by the microelectronic nonlinear unit, encode the nonlinear characteristics in the electrical signal back into the optical field through the electro-optic effect, and output the optical field signal.

[0020] In one embodiment, the nonlinear activation module is a module constructed using all-optical nonlinear materials.

[0021] In one embodiment, the nonlinear activation module includes a switching unit, a first nonlinear activation unit, and a second nonlinear activation unit;

[0022] The switching unit is connected to the first nonlinear activation unit and the second nonlinear activation unit respectively. The switching unit is used to control the target nonlinear activation unit as a working activation unit on the optical signal transmission path according to the input command. The target nonlinear activation unit is one of the first nonlinear activation unit and the second nonlinear activation unit.

[0023] The first nonlinear activation unit includes a photodetector, a microelectronic nonlinear unit, and an electro-optic modulator; the photodetector is used to convert the standard optical input signal after matrix multiplication into an electrical signal; the electro-optic modulator is used to receive the electrical signal processed by the microelectronic nonlinear unit, encode the nonlinear features in the electrical signal back into the optical field through the electro-optic effect, and output the optical field signal.

[0024] The second nonlinear activation unit is a unit constructed using an all-optical nonlinear material.

[0025] In one embodiment, the nonlinear activation module is provided between adjacent optical modulation arrays.

[0026] In one embodiment, there are multiple weight configuration modules, which are cascaded sequentially; the nonlinear activation module is provided between adjacent weight configuration modules.

[0027] In one embodiment, the optical modulation array is an MZI array or a waveguide array; in the cascaded path of the multi-level weight configuration module, the waveguide intersection adopts an adiabatic transition structure or a subwavelength grating.

[0028] In one embodiment, the optical neural network computing layer further includes an optical power monitor and an electronic control unit;

[0029] The optical power monitor is used to receive and measure the optical field signal output by the optical neural network computing layer in real time, and feed the measurement data back to the electronic control unit;

[0030] The electronic control unit is used to dynamically adjust the driving voltage to achieve the thermo-optical effect or electro-optical effect based on the measurement data.

[0031] The integrated display system based on optical neural networks proposed in this application has the following advantages over related technologies:

[0032] 1. The integrated display system based on optical neural networks in this application directly senses external environmental signals and modulates the input light signals to form a standard light input signal through the sensing layer. This eliminates the redundant conversion of "physical signal to electrical signal" at the sensing end in the traditional architecture, and realizes the synchronous completion of environmental signal reception and light input encoding. The optical neural network computing layer performs matrix multiplication operations in parallel in the optical domain and completes nonlinear activation. Compared with the electrical domain processing of traditional electronic computing chips, it greatly improves the computing efficiency and avoids the transmission overhead of electrical signals in the computing stage. At the same time, through the low-loss light output structure of optical waveguide structure or microlens group, the output light field signal is directly coupled to each display unit of the display layer, reducing the loss and delay of signal transmission between modules.

[0033] 2. In this application, the display unit only uses a photoelectric conversion structure to convert the coupled light signal into a driving electrical signal to control the light emission brightness. No additional display driving circuit is required, which further simplifies the link process and ultimately realizes efficient transmission and processing of the entire link from environmental signal perception to visualization output. This significantly reduces system energy consumption and latency and improves integration.

[0034] 3. The optical power monitor receives and measures the optical field signal output by the optical neural network computing layer in real time, and feeds the measurement data back to the electronic control unit. Then, based on the measurement data, the electronic control unit dynamically adjusts the driving voltage to realize the thermo-optic effect or electro-optic effect. In combination with the closed-loop optical power monitoring and online fine-tuning mechanism, it effectively compensates for errors caused by environmental interference, temperature drift and other factors, and ensures the accuracy and stability of the system in long-term operation. It is especially suitable for high-bandwidth, large-data-volume real-time image processing or AI inference scenarios, and perfectly solves the performance bottleneck of traditional architecture in such scenarios. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of the structure of an integrated display system based on an optical neural network in one embodiment of this application;

[0037] Figure 2 This is a schematic diagram of the structure of an optical neural network in one embodiment of this application;

[0038] Figure 3 This is a schematic diagram of the structure of an integrated display system based on an optical neural network in another embodiment of this application.

[0039] Explanation of reference numerals in the attached figures:

[0040] 10-Integrated display system based on optical neural network, 11-Sensing layer, 111-Photosensitive sensor array, 112-Coupled structure, 113-Optical modulator, 12-Optical neural network computing layer, 13-Display layer, 131-Photoelectric conversion structure, 132-Light emission control structure. Detailed Implementation

[0041] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0042] It should be noted that when a component is said to be "fixed to" another component, it can be directly attached to the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0043] In the following embodiments, when a layer, region, or element is “connected,” it can be interpreted as the layer, region, or element being connected not only directly but also through other constituent elements placed therebetween. For example, when a layer, region, element, etc., is described as being connected or electrically connected, the layer, region, element, etc., can not only be directly connected or directly electrically connected, but can also be connected or electrically connected through another layer, region, element, etc., placed therebetween.

[0044] In the following text, although terms such as “first” and “second” may be used to describe various components, these components are not necessarily limited to the terms above. The terms above are only used to distinguish one component from another. It will also be understood that expressions used in the singular form include plural expressions, unless the singular form has a distinctly different meaning in the context.

[0045] As used in the application documents, it should also be understood that the terms "comprising / including" or "having" specify the presence of the stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof.

[0046] In some embodiments, such as Figure 1 As shown, this application provides an integrated display system 10 based on optical neural networks, including: a perception layer 11, an optical neural network computing layer 12, and a display layer 13.

[0047] The perception layer 11 is used to sense external environmental signals and modulate the input light signal according to the external environmental signals to form a standard light input signal for the corresponding optical neural network. External environmental signals can include at least one of signals such as ambient light, pressure, fingerprints, micro-vibrations, and target reflection patterns. The perception layer 11 first captures key physical signals in the external environment in real time through an integrated sensor array, eliminating the redundant "physical signal → electrical signal" conversion required in traditional architectures and directly using the environmental signals as the modulation basis. Subsequently, based on the characteristics of the captured environmental signals, the transmitted input light signal is precisely controlled. Specifically, differentiated modulation of the phase, amplitude, or polarization state of the light signal can be achieved, creating a one-to-one mapping between the optical characteristics of the light signal and the physical parameters of the environmental signals. Finally, the modulated light signal is normalized into a standard light input vector that meets the parallel operation requirements of the optical neural network computing layer 12, i.e., the standard light input signal.

[0048] The optical neural network computation layer 12 receives the standard optical input signal through an optical neural network with preset weight parameters. Based on the modulated optical signal, it performs matrix multiplication operations in parallel within the optical domain and performs nonlinear activation to output an optical field signal. The optical field signal is then coupled to each display unit of the display layer 13 through an optical output structure, which can be an optical waveguide structure or a microlens group. The structure of the optical neural network can be found in [reference needed]. Figure 2 .

[0049] The optical neural network computing layer 12 incorporates a built-in optical neural network with preset weight parameters. This layer first receives a standard optical input signal from the perception layer 11, and then performs large-scale parallel matrix multiplication operations synchronously in the optical domain based on the modulated optical signal. Compared to the serial operation mode of traditional electronic computing chips, optical domain parallel operation can overcome the bandwidth bottleneck of electronic devices, significantly improving the processing efficiency of high-dimensional data, while avoiding signal transmission delay and energy loss during electrical domain operations. After completing the linear operation of matrix multiplication, the computing layer performs nonlinear mapping on the processed optical signal through an integrated nonlinear activation module. Nonlinear activation effectively extracts depth features from the optical signal, preventing the network from degenerating into a simple linear model. Finally, the nonlinearly activated optical field signal is precisely coupled to each display unit of the display layer 13 through two low-loss optical output structures: an optical waveguide structure or a microlens group. The optical waveguide structure enables low crosstalk and long-distance transmission of the optical signal, adapting to high-density integration scenarios; the microlens group can precisely map the optical field signal to the corresponding display unit pixel location through focusing, ensuring a one-to-one correspondence between the optical field signal and the display unit. The entire process is a full-optical-domain processing link, eliminating redundant steps in electrical signal processing and conversion in traditional architectures, and significantly improving system computing efficiency and integration.

[0050] The display unit includes a photoelectric conversion structure 131 and a light-emitting control structure 132. The photoelectric conversion structure 131 converts the received coupled optical signal into a driving electrical signal and sends it to the light-emitting control structure 132 to control the brightness of the display unit. The light-emitting control structure 132 is a liquid crystal cell or a light-emitting element. The photoelectric conversion structure 131 can employ a high-response photoelectric detection element, such as a quantum dot photodiode or an integrated PIN photodetector.

[0051] It can be understood that the display unit is the visualization output terminal of the integrated display system. It achieves precise mapping from light field signals to visual brightness output through the coordinated action of the photoelectric conversion structure 131 and the light-emitting control structure 132. Specifically, the photoelectric conversion structure 131 receives light field signals from the optical neural network computing layer 12, precisely coupled through an optical waveguide structure or microlens group. Without the complex signal format conversion of traditional architectures, it directly converts the optical characteristics of the light field signal, such as intensity and phase distribution, into corresponding driving electrical signals through the photovoltaic effect or photoconductive effect. The amplitude, frequency, and other parameters of this electrical signal strictly match the calculation results carried by the light field signal, ensuring the fidelity and real-time performance of the signal conversion. Subsequently, this driving electrical signal is transmitted in real-time to the light-emitting control structure 132. If the light-emitting control structure 132 is a liquid crystal unit, the driving electrical signal will adjust the orientation state of the liquid crystal molecules, changing the unit's transmittance, thereby achieving continuous adjustment of the display brightness to meet the display requirements of high resolution and low power consumption. If it is a light-emitting element (such as a micro LED, OLED device, etc.), the driving electrical signal directly excites the element to emit light. The brightness of the light emission has a linear or non-linear relationship with the intensity of the electrical signal, which can meet the display scenarios of high contrast and fast response.

[0052] The aforementioned integrated display system 10 based on optical neural networks directly senses external environmental signals and modulates the input light signal to form a standard light input signal through the sensing layer 11. This eliminates the redundant conversion between "physical signal" and "electrical signal" at the sensing end in traditional architectures, enabling simultaneous completion of environmental signal reception and light input encoding. The optical neural network computing layer 12 performs matrix multiplication operations in parallel within the optical domain and completes nonlinear activation. Compared to the electrical domain processing of traditional electronic computing chips, this significantly improves computational efficiency and avoids the transmission overhead of electrical signals in the computation stage. Simultaneously, through a low-loss light output structure such as an optical waveguide structure or a microlens group, the output light field signal is directly coupled to each display unit of the display layer 13, reducing signal transmission loss and delay between modules. The display unit only needs to convert the coupled light signal into a driving electrical signal through the photoelectric conversion structure 131 to control the light emission brightness, eliminating the need for additional display driving circuits. This further simplifies the link process, ultimately achieving efficient transmission and processing across the entire link from environmental signal perception to visual output, significantly reducing system energy consumption and latency, and improving integration.

[0053] In some embodiments, such as Figure 3 As shown, the sensing layer 11 includes: a photosensitive sensing array 111, a coupling structure 112, and a light modulator 113.

[0054] The coupling structure 112 is used to couple external incident light into the input waveguide corresponding to the photosensitive sensing array 111. The photosensitive sensing array 111 is used to change its refractive index or absorptivity according to the external environmental signal to modulate the input optical signal transmitted in the input waveguide. The optical modulator 113 is used to encode the optical signal modulated by the photosensitive sensing array 111 to form a standard optical input signal.

[0055] The coupling structure 112 can be an optical structure such as a low-loss grating coupler, a high-precision end-face coupler, or a free-space incident structure (e.g., a microlens array). The photosensitive sensing array 111 can be a high-sensitivity photosensitive element such as a quantum dot photosensitive unit, a two-dimensional material (e.g., graphene, MoS2) sensing array, or a semiconductor quantum well sensing structure.

[0056] It can be understood that the coupling structure 112 is a key connecting component between the external incident light and the input waveguide. It can efficiently converge and couple the external parallel or divergent light to the input waveguide corresponding to the photosensitive sensing array 111 with low loss, minimizing energy loss and mode distortion of the optical signal during the coupling process, and ensuring the intensity stability and transmission efficiency of the input optical signal. The photosensitive sensing array 111 is arranged in an array format, corresponding one-to-one with the input waveguide. When an external environmental signal acts on the array, the photosensitive sensing array 111 will dynamically change its refractive index or absorptivity through mechanisms such as photogenerated carrier injection, stress-induced lattice distortion, or thermo-optical effects, thereby precisely modulating the optical signal transmitted in the input waveguide. Specifically, it can realize continuous phase shift, gradient attenuation of amplitude, or directional conversion of polarization state of the optical signal, so that the optical characteristics of the optical signal and the physical parameters of the external environmental signal form a precise mapping relationship.

[0057] As the core unit of signal encoding, the optical modulator 113 integrates an electro-optic modulation array or an all-optical modulation structure. It receives the optical signal that has been initially modulated by the photosensitive sensing array 111, and performs secondary normalization and formatting processing on the phase, amplitude or polarization state of the optical signal through preset encoding rules. This may include operations such as phase calibration, amplitude normalization, and channel synchronization alignment. Finally, it encodes the signal into a standard optical input signal that meets the parallel operation requirements of the optical neural network computing layer 12, ensuring that the standard optical input signal has a unified input format, a stable optical field distribution and clear feature recognition.

[0058] In some embodiments, such as Figure 2 As shown, the optical neural network computation layer 12 includes: a weight configuration module, a nonlinear activation module, and an output module.

[0059] The weight configuration module includes a multi-level optical modulation array. This module performs matrix multiplication on the standard optical input signal and modulates the phase of the optical modulation array through electro-optic or thermo-optic effects to achieve weight configuration in the matrix multiplication operation. The optical modulation array can be one of an MZI array, a metasurface array, or a waveguide array. The nonlinear activation module performs nonlinear activation on the standard optical input signal after matrix multiplication, outputting an optical field signal. The output module couples the optical field signal to each display unit of the display layer 13 through an optical output structure.

[0060] The weight configuration module, serving as the core of linear computation, integrates multi-level optical modulation arrays. Nonlinear activation modules are placed between adjacent optical modulation array levels to ensure that each array is arranged in an alternating logical sequence of "linear computation - nonlinear activation," supporting complex deep learning inference functions. The modulated optical signal enters the pre-configured weighted optical neural network (ONN) core module, which is typically composed of an MZI array, metasurface, or waveguide array, and is used to perform matrix-vector multiplication: Y = WX + B.

[0061] In applications, the phase of MZI or waveguides can be precisely controlled through electro-optic or thermo-optic effects to configure the weights W of a neural network. Specifically, MZI arrays rely on the phase difference of waveguide arms to enhance or attenuate interference to map weights, waveguide arrays complete weighted summation by controlling the phase of the coupling strength between adjacent waveguides, and metasurface arrays utilize diffraction phase modulation of subwavelength structures to achieve parallel weighting, ultimately completing matrix multiplication operations efficiently in the optical domain.

[0062] As a key to overcoming the limitations of linear computation, the nonlinear activation module can flexibly adopt O / E / O type activation units and / or all-optical nonlinear activation units according to task requirements. The nonlinear activation module performs nonlinear mapping on the linear computation results output by each level of weight configuration module, effectively extracting depth features in the signal, avoiding network degradation into a simple linear model, and finally outputting an optical field signal with complex feature representation.

[0063] The output module is responsible for the low-loss transmission and precise coupling of the light field signal. Through the optical waveguide structure or microlens group, the nonlinearly activated light field signal is efficiently coupled to the corresponding display unit according to the spatial arrangement of each display unit in the display layer 13, so as to achieve seamless connection between the optical domain calculation results and the display terminal. The entire calculation layer significantly reduces the calculation latency and energy consumption through the design of optical domain parallel operation and redundancy-free cross-domain conversion, while improving the system integration and perfectly adapting to real-time processing scenarios with high bandwidth and large data volume.

[0064] In some embodiments, the nonlinear activation module includes a photodetector, a microelectronic nonlinear unit, and an electro-optic modulator 113.

[0065] The photodetector is used to convert the standard optical input signal after matrix multiplication into an electrical signal. The electro-optic modulator 113 is used to receive the electrical signal after processing by the microelectronic nonlinear unit, encode the nonlinear characteristics in the electrical signal back into the optical field through the electro-optic effect, and output the optical field signal.

[0066] Among them, the photodetector can be an integrated PIN photodiode or an avalanche photodiode (APD) with high responsivity and high bandwidth.

[0067] In this embodiment, the nonlinear activation module adopts an O / E / O type activation unit adapted to high-precision scenarios, including a photodetector, a microelectronic nonlinear unit, and an electro-optic modulator 113. The photodetector, the microelectronic nonlinear unit, and the electro-optic modulator 113 are connected in series according to an "optical-electrical-optical" signal conversion link to achieve accurate mapping from linear calculation results to nonlinear characteristic light field signals. The electro-optic modulator 113 can be a high-performance device such as a lithium niobate (LN) modulator or a silicon-based electro-optic modulator 113.

[0068] After receiving the optical signal from the weighted configuration module after matrix multiplication, the photodetector efficiently converts the optical characteristics such as intensity and phase of the optical signal into an electrical signal with a linear relationship through photogenerated carrier injection or photoconductive effect. Then, the microelectronic nonlinear unit, as the core of the electrical domain nonlinear processing, integrates dedicated analog circuits or programmable logic units, and has built-in hardware implementation logic for commonly used deep learning activation functions such as ReLU and Sigmoid. Through operational amplifiers, nonlinear resistor networks, or FPGA programmable logic arrays, it performs nonlinear mapping processing on the electrical signal output by the photodetector, accurately extracts the depth features in the signal, and has the function of dynamically adjusting the activation threshold and slope to adapt to the needs of different deep learning tasks. The electro-optic modulator 113 receives the nonlinear electrical signal processed by the microelectronic nonlinear unit through electrodes, and uses the electro-optic effect to convert the voltage change of the electrical signal into a dynamic adjustment of its own refractive index, thereby modulating the phase, amplitude, or intensity of the input carrier optical signal. Finally, it accurately encodes the nonlinear characteristics of the electrical domain back into the optical field and outputs an optical field signal with complex characteristics. This signal strictly matches the transmission requirements of the output module to ensure efficient connection with the subsequent optical output structure and display layer 13. The entire module uses a closed-loop conversion link of "optical signal → electrical signal → nonlinear electrical signal → optical field signal" to retain the high-precision advantages of electrical domain processing and achieve seamless compatibility with optical domain operations in the 12th optical neural network computing layer. This effectively avoids the network from degenerating into a simple linear model and provides stable nonlinear feature extraction capabilities for complex AI inference tasks.

[0069] In other embodiments, the nonlinear activation module is a module constructed using all-optical nonlinear materials.

[0070] In this embodiment, the nonlinear activation module is an all-optical activation unit adapted to ultra-low power consumption and high-speed response scenarios. The nonlinear activation module can be constructed based on all-optical nonlinear materials such as graphene, semiconductor quantum dots, lithium niobate, and semiconductor quantum wells. It does not rely on cross-domain conversion components such as photodetectors, microelectronic nonlinear units, or electro-optic modulators 113, and achieves nonlinear mapping of signals entirely by relying on the inherent optical nonlinear effects of the materials themselves.

[0071] When the nonlinear activation module is constructed using all-optical nonlinear materials, changes in optical parameters such as light intensity and phase when an optical signal acts on it will induce nonlinear responses in the material's refractive index and absorptivity. For example, the absorption coefficient of a saturated absorbing material drops sharply under high light intensity, the refractive index of a Kerr material changes nonlinearly with light intensity, and a two-photon absorbing material achieves energy level transitions by absorbing two photons, thereby nonlinearly modulating the phase, amplitude, or frequency of the incident light signal. This completes the transformation from linear computation results to complex feature representations, ultimately outputting a light field signal with nonlinear characteristics. The entire process does not require a cross-domain conversion between "optical-electrical-optical," eliminating redundant steps in electrical domain processing and significantly reducing energy loss and signal delay.

[0072] In some other embodiments, the nonlinear activation module includes a switching unit, a first nonlinear activation unit, and a second nonlinear activation unit.

[0073] The switching unit is connected to the first nonlinear activation unit and the second nonlinear activation unit respectively. The switching unit is used to control the target nonlinear activation unit as the working activation unit on the optical signal transmission path according to the input command. The target nonlinear activation unit is one of the first nonlinear activation unit and the second nonlinear activation unit.

[0074] The first nonlinear activation unit includes a photodetector, a microelectronic nonlinear unit, and an electro-optic modulator 113; the photodetector is used to convert the standard optical input signal after matrix multiplication into an electrical signal; the electro-optic modulator 113 is used to receive the electrical signal processed by the microelectronic nonlinear unit, encode the nonlinear characteristics in the electrical signal back into the optical field through the electro-optic effect, and output the optical field signal.

[0075] The second nonlinear activation unit is a unit constructed using all-optical nonlinear materials.

[0076] The switching unit uses low crosstalk and fast response optical switching devices, such as MZI type electro-optic switches, microelectromechanical systems (MEMS) optical switches or thermo-optic switches.

[0077] It is understandable that the switching unit, as the control core for mode switching, can establish independent optical signal paths with the first and second nonlinear activation units respectively through waveguides or optical fiber links. It can quickly switch the optical signal transmission path according to external input commands, precisely selecting either the first or second nonlinear activation unit as the sole working activation unit. The first nonlinear activation unit is an O / E / O type activation structure adapted for high-precision scenarios, while the second nonlinear activation unit is an all-optical activation structure adapted for ultra-low power consumption and high-speed response scenarios. Through the complementary design of the two activation units and the flexible switching of the switching unit, the entire module can selectively ensure the processing quality of high-precision tasks or meet the efficiency requirements of low-power scenarios. Simultaneously, the output optical field signals of both modes can be efficiently adapted to the optical output structure of subsequent output modules, ensuring the continuity and compatibility of the entire processing chain of the optical neural network computing layer 12.

[0078] It should be noted that the first nonlinear activation unit and the second nonlinear activation unit provided in the embodiments of this application can refer to the description in the foregoing embodiments, and the repeated parts will not be described again.

[0079] In some embodiments, there are multiple weight configuration modules, which are cascaded sequentially; a non-linear activation module is provided between adjacent weight configuration modules.

[0080] It can be understood that multiple cascaded weight configuration modules each carry an independent optical control array. Each module pre-controls the array phase distribution through electro-optic or thermo-optic effects, solidifies the preset weight parameters of the corresponding level, and after receiving the optical signal output from the previous level, performs matrix multiplication operations in parallel within the optical domain to achieve linear transformation and dimensional mapping of features. The nonlinear activation module between adjacent levels performs nonlinear mapping processing on the linear operation results output by the previous level weight configuration module. By extracting high-order features from the signal, it avoids the network from degenerating into a simple linear model, ensuring that the features of each level can be effectively abstracted and optimized. The processed optical field signal is then passed to the next level weight configuration module to achieve layer-by-layer feature deepening. This multi-layered cascaded and non-linear activation interleaving design perfectly replicates the core logic of multi-layer fully connected networks in deep learning. Compared with traditional single-layer linear operations, it significantly improves the system's ability to model and process complex data. At the same time, relying on the natural advantages of optical domain parallel computing, it avoids the cumulative latency and energy consumption caused by multi-layer electrical domain processing, further enhancing the system's real-time processing performance in high-bandwidth, high-data-volume scenarios. It ensures that the entire link from standard optical input signal to final optical field output can efficiently complete the extraction and transformation of complex features, providing accurate visualization data support for display layer 13.

[0081] In some embodiments, the optical modulation array is an MZI array or a waveguide array; in the cascaded path of the multi-level weight configuration module, the waveguide intersection adopts an adiabatic transition structure or a subwavelength grating.

[0082] It is understandable that when the optical control array is an MZI array, the MZI array can achieve weighted summation of signals by adjusting the phase difference between the two waveguide arms, utilizing the interference effect of light, thereby completing the weight mapping in matrix multiplication. When the optical control array is a waveguide array, the waveguide array constructs parallel optical signal channels through the adjustment of coupling strength and phase calibration between adjacent waveguides, efficiently performing large-scale matrix operations. Both can precisely configure weight parameters through electro-optic or thermo-optic effects, ensuring the accuracy and flexibility of linear operations.

[0083] In a cascaded path of multiple weighted configuration modules, since both the MZI array and the waveguide array rely on waveguides for signal transmission, waveguide crossings are inevitable in the optical signal interaction between layers. These crossings can easily lead to mode coupling, signal crosstalk, and energy loss. Therefore, adiabatic transition structures or subwavelength gratings are specifically designed for optimization. The adiabatic transition structure, through its gradually varying waveguide width and coupling region design, allows for a smooth transition of optical signals at the crossing points, minimizing mode distortion and energy leakage. The subwavelength grating, utilizing its super-diffraction-limited optical field manipulation capabilities, achieves optical signal isolation and low-loss transmission between crossing waveguides through specific periodic arrangements and refractive index distributions. Both effectively suppress crosstalk and reduce transmission loss, ensuring the fidelity and transmission efficiency of optical signals during multi-stage cascading.

[0084] In some embodiments, the optical neural network computing layer 12 further includes an optical power monitor and an electrical control unit.

[0085] The optical power monitor is used to receive and measure the optical field signal output by the optical neural network computing layer 12 in real time, and feed the measurement data back to the electronic control unit.

[0086] The electronic control unit is used to dynamically adjust the driving voltage to achieve the thermo-optical effect or electro-optical effect based on measurement data.

[0087] Among them, the optical power monitor can be a high-sensitivity integrated optical power meter or an arrayed photoelectric detection unit.

[0088] It is understandable that the optical power monitor and the electronic control unit (ECU) work together to construct a high-precision closed-loop feedback control mechanism, providing crucial guarantees for the long-term stability and weight configuration accuracy of optical domain computation. The optical power monitor, deployed at the front end of the optical output structure in the computing layer, can receive and accurately measure the final output optical field signal after nonlinear activation in real time, acquiring core parameters such as the absolute value of optical power, power distribution uniformity, and power consistency of each channel. This quantified measurement data is then fed back to the ECU in real time as an electrical signal. The ECU, as the control center of the entire closed-loop system, has a built-in preset standard optical power threshold and optical field distribution model. By comparing and analyzing the feedback measurement data with the standard model, it quickly identifies optical field signal deviations caused by factors such as ambient temperature drift, device process deviations, and long-term aging. Based on a deviation compensation algorithm, it then dynamically and precisely adjusts the driving voltage used to achieve thermo-optical or electro-optical effects.

[0089] For the thermo-optical modulation path, the electronic control unit adjusts the supply voltage of the micro-heater and corrects the refractive index by changing the waveguide temperature, thereby compensating for phase and optical power. For the electro-optical modulation path, the electronic control unit adjusts the driving voltage of the electro-optic modulator 113 or the MZI array electrodes, directly calibrating the phase or amplitude of the optical signal using electro-optic effects such as the Pockels effect. This corrects weight configuration errors and ensures that the optical field signal output by the optical neural network computing layer 12 always maintains stable power and distribution characteristics. The entire closed-loop modulation process requires no manual intervention, featuring fast response speed and high adjustment accuracy. It effectively offsets the negative impacts of external interference and internal device drift, significantly improving the computational robustness and long-term operational reliability of the optical neural network computing layer 12.

[0090] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An integrated display system based on an optical neural network, characterized in that, include: The sensory layer, the optical neural network computing layer, and the display layer; The sensing layer includes: a photosensitive sensor array, a coupling structure, and an optical modulator; the coupling structure is used to couple external incident light to the input waveguide corresponding to the photosensitive sensor array; the photosensitive sensor array is used to change its refractive index or absorptivity according to the external environmental signal to modulate the input optical signal transmitted in the input waveguide; the optical modulator is used to encode the optical signal modulated by the photosensitive sensor array to form a standard optical input signal; The optical neural network computing layer includes a weight configuration module, a nonlinear activation module, and an output module. The weight configuration module includes a multi-level optical modulation array, which performs matrix multiplication on the standard optical input signal and modulates the phase of the optical modulation array through electro-optic or thermo-optic effects to achieve weight configuration in the matrix multiplication operation. The nonlinear activation module performs nonlinear activation on the standard optical input signal after matrix multiplication and outputs a light field signal. The output module couples the light field signal to each display unit of the display layer through a light output structure. The optical modulation array is one of an MZI array, a metasurface array, and a waveguide array, and the light output structure is a waveguide structure or a microlens group. The display unit includes a photoelectric conversion structure and a light-emitting control structure. The photoelectric conversion structure is used to convert the received coupled optical signal into a driving electrical signal and send it to the light-emitting control structure to control the light-emitting brightness of the display unit. The light-emitting control structure is a liquid crystal cell or a light-emitting element.

2. The integrated display system based on optical neural networks as described in claim 1, characterized in that, The nonlinear activation module includes a photodetector, a microelectronic nonlinear unit, and an electro-optic modulator; The photodetector is used to convert the standard optical input signal after matrix multiplication into an electrical signal. The electro-optic modulator is used to receive the electrical signal processed by the microelectronic nonlinear unit, encode the nonlinear characteristics in the electrical signal back into the optical field through the electro-optic effect, and output the optical field signal.

3. The integrated display system based on optical neural networks as described in claim 1, characterized in that, The nonlinear activation module is constructed using all-optical nonlinear materials.

4. The integrated display system based on optical neural networks as described in claim 1, characterized in that, The nonlinear activation module includes a switching unit, a first nonlinear activation unit, and a second nonlinear activation unit. The switching unit is connected to the first nonlinear activation unit and the second nonlinear activation unit respectively. The switching unit is used to control the target nonlinear activation unit as a working activation unit on the optical signal transmission path according to the input command. The target nonlinear activation unit is one of the first nonlinear activation unit and the second nonlinear activation unit. The first nonlinear activation unit includes a photodetector, a microelectronic nonlinear unit, and an electro-optic modulator; the photodetector is used to convert the standard optical input signal after matrix multiplication into an electrical signal; the electro-optic modulator is used to receive the electrical signal processed by the microelectronic nonlinear unit, encode the nonlinear features in the electrical signal back into the optical field through the electro-optic effect, and output the optical field signal. The second nonlinear activation unit is a unit constructed using an all-optical nonlinear material.

5. The integrated display system based on optical neural networks as described in claim 1, characterized in that, The nonlinear activation module is provided between adjacent optical modulation arrays.

6. The integrated display system based on optical neural networks as described in claim 1, characterized in that, The number of weight configuration modules is multiple, and the multiple weight configuration modules are cascaded in sequence; the nonlinear activation module is provided between adjacent weight configuration modules.

7. The integrated display system based on optical neural networks as described in claim 6, characterized in that, The optical modulation array is an MZI array or a waveguide array; in the cascaded path of the multi-level weight configuration module, the waveguide intersection adopts an adiabatic transition structure or a subwavelength grating.

8. The integrated display system based on optical neural networks as described in claim 1, characterized in that, The optical neural network computing layer also includes an optical power monitor and an electronic control unit; The optical power monitor is used to receive and measure the optical field signal output by the optical neural network computing layer in real time, and feed the measurement data back to the electronic control unit; The electronic control unit is used to dynamically adjust the driving voltage to achieve the thermo-optical effect or electro-optical effect based on the measurement data.

Images