Image processing system and method based on wavelength parameter reconfigurable optical neural network
By reconstructing optical neural networks using wavelength parameters, electronic parameters can be transferred to optical wavelength parameters, thereby reconstructing the computing functions of passive diffraction chips. This solves the problem of rigid optical neural network structures and enables efficient and flexible optical computing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2025-12-02
- Publication Date
- 2026-04-17
AI Technical Summary
Existing optical neural network chips have a fixed structure, making them unable to adapt to different tasks or model updates. The reconstruction process is complex and inefficient, which limits their flexibility and practicality.
By using a reconfigurable optical neural network based on wavelength parameters, traditional electronic parameters are transferred to optical wavelength parameters. Wavelength switching is used to reconfigure the computing function of a passive diffraction chip, and the data cache module and optical weight storage module are coordinated to achieve passive modulation and fast switching of network or computing layers.
Without increasing external hardware resources, it achieves a leapfrog improvement in computing speed and energy efficiency, with the advantages of low power consumption, high parallelism and light-speed processing, overcoming the energy efficiency and bandwidth limitations of traditional electronic computing.
Smart Images
Figure CN121882141A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to an image processing system and method based on a wavelength parameter reconfigurable optical neural network. Background Technology
[0002] With the rapid development of artificial intelligence technology, deep neural networks have become a core driving force in fields such as machine vision and autonomous driving. However, the rise of neural network models, especially large-scale models, has posed unprecedented challenges to the computing power, energy efficiency, and parallel processing capabilities of computing platforms. Traditional electronic computing platforms based on the von Neumann architecture, limited by the failure of Moore's Law, bandwidth bottlenecks between storage and processing units, and ever-increasing energy consumption, are struggling to meet the demands of future high-parallelism, high-throughput AI computing, becoming a key bottleneck restricting the development of AI.
[0003] To overcome the bottlenecks of traditional electronic computing, optical computing architectures based on analog computing and neuromorphic photonics, especially optical neural networks, have attracted much attention due to their potential for energy efficiency, bandwidth, and parallel processing at the speed of light. Among them, on-chip optical computing architectures, compared with interferometric and wavelength division computing architectures, diffraction-based schemes can integrate a large number of connections and parameters onto optical structures by taking advantage of the broadcasting characteristics of light diffraction, achieving extremely high computational parallelism and passive computing processes. They are regarded as an effective way to solve the energy efficiency and speed bottlenecks of electronic computing.
[0004] However, the structure of diffraction-based optical neural network chips is fixed and difficult to reconfigure. This means that once the chip is manufactured, the neural network functions and parameters are fixed and cannot be adapted to different tasks or updated. The reconfiguration schemes proposed so far not only require the introduction of additional optical and electronic hardware, increasing system cost and complexity, but also consume auxiliary computing resources, resulting in a complex and inefficient reconfiguration process. These shortcomings severely restrict the flexibility and practicality of optical neural networks, making it difficult for them to fully realize their inherent advantages of high speed and high energy efficiency. Summary of the Invention
[0005] This invention provides an image processing system and method based on wavelength parameter reconfigurable optical neural networks. It addresses the shortcomings of existing technologies where the fixed structure necessitates additional hardware and auxiliary computing resources for reconstruction, resulting in a complex and inefficient process. This invention can transfer traditional electronic parameters to optical wavelength parameters without increasing external hardware computing resources or additional reconstruction algorithms, thereby fully leveraging optical advantages. Furthermore, it enables the reconstruction of passive diffraction chip computing functions through wavelength switching, overcoming the limitations of traditional electronic computing in terms of energy efficiency and bandwidth. It offers significant advantages such as low power consumption, high parallelism, and light-speed processing.
[0006] This invention provides an image processing system based on a wavelength parameter reconfigurable optical neural network, comprising an optical image processing module, a synchronization module, an optical weight storage module, and a data cache module. The synchronization module sends synchronization signals to both the optical weight storage module and the data cache module. The data cache module sends cached data to the optical image processing module based on the synchronization signals. The cached data characterizes the initial input image to be processed or the intermediate image processing result cached by the optical image processing module. The optical weight storage module sends corresponding wavelength parameters and corresponding response parameters to the optical image processing module based on the synchronization signals. The wavelength parameters and corresponding response parameters are trained and stored based on a previously constructed photonic deep convolutional neural network. The optical image processing module passively modulates the input cached data based on the input wavelength parameters and corresponding response parameters to obtain the corresponding probe light intensity, and outputs the probe light intensity as the final result or updates the data cache module as an intermediate image processing result.
[0007] According to the present invention, an image processing system based on a wavelength parameter reconfigurable optical neural network is provided. The optical image processing module includes an encoding unit, a modulation unit, a diffraction core unit, and a detection unit. The encoding unit encodes buffered data to obtain an encoded optical signal. The modulation unit modulates the encoded optical signal according to the wavelength parameter, modulates the corresponding optical signal according to the response parameter corresponding to the wavelength parameter, and transmits the modulated wavelength division multiplexing optical signal to the diffraction core unit via a waveguide. The diffraction core unit utilizes a subwavelength structure to perform matrix calculations on the wavelength division multiplexing optical signal input through the waveguide to obtain different light field distributions at different spatial locations. The detection unit detects the light field distribution output by the diffraction core unit to obtain the corresponding detection light intensity.
[0008] According to the present invention, an image processing system based on a wavelength parameter reconfigurable optical neural network is provided. The modulation unit is used to transmit coded optical signals through a main waveguide. The modulation unit includes at least one micro-ring modulator, which is used to: couple the optical signal of the corresponding wavelength in the coded optical signal passing through the coupling region of the micro-ring modulator in the main waveguide to the micro-ring resonant cavity of the micro-ring modulator according to the received wavelength parameter; and perform amplitude modulation on the optical signal of the corresponding wavelength in the micro-ring resonant cavity based on the response parameter corresponding to the wavelength parameter, and release the modulated optical signal into the main waveguide.
[0009] According to the image processing system based on a wavelength parameter reconfigurable optical neural network provided by the present invention, the number of micro-ring modulators is determined in advance according to the kernel size of the photonic deep convolutional neural network; when the number of micro-ring modulators is greater than one, for each micro-ring modulator, the corresponding wavelength parameter and the response parameter corresponding to the wavelength parameter are received respectively, and the micro-ring modulators correspond one-to-one with the wavelength parameters; for each micro-ring modulator, the corresponding optical signal is independently modulated, and the obtained wavelength division multiplexed signal is transmitted to the same waveguide.
[0010] According to the image processing system based on a wavelength parameter reconfigurable optical neural network provided by the present invention, the subwavelength structure is configured in advance based on different wavelength transformation methods. The different wavelength transformation methods are determined in advance based on the wavelength response spectrum obtained by scanning the spectrum of an optical image processing module without modulation and detection units. The diffraction core unit is used to project the light signal of the corresponding wavelength onto the corresponding spatial position of the output plane through the subwavelength structure according to the transformation method of each wavelength. The detection unit includes a photodetector array, and the number of photodetectors in the photodetector array is determined in advance according to the number of channels of the photonic deep convolutional neural network.
[0011] An image processing system based on a wavelength parameter reconfigurable optical neural network according to the present invention further includes a training module, specifically comprising: a spectral scanning unit for performing spectral scanning measurements on an optical image processing module without a modulation unit and a detection unit to obtain the wavelength response spectral lines of each output port; a network construction unit for constructing an all-optically parameterized photonic deep convolutional neural network based on the wavelength response spectral lines of each output port; a training data acquisition unit for acquiring image training data and the ground truth values corresponding to the image training data; and a training unit for training the photonic deep convolutional neural network by using the image training data as input data for training and the ground truth values corresponding to the image training data as labels for training.
[0012] According to the image processing system based on a wavelength parameter reconfigurable optical neural network provided by the present invention, the network construction unit is further configured to: determine the response parameters corresponding to the wavelength of each channel according to the wavelength response spectrum of each output port, and construct an optical weight pool; the response parameters are used to characterize the transmission characteristics of the corresponding channel, and the channel is used to characterize the transmission path of the wavelength from the input optical image processing module to the output; according to any wavelength combination, find the response parameters of the corresponding wavelength from the optical weight pool and map them to parallel multi-convolution kernel weights to construct a photonic deep convolutional neural network; the number of wavelength parameters in the wavelength combination is configured according to the number of micro-ring modulators in the modulation unit.
[0013] According to the image processing system based on a wavelength parameter reconfigurable optical neural network provided by the present invention, the training unit is further configured to train the photonic deep convolutional neural network according to a preset iterative strategy, and determine the response parameters corresponding to each convolutional kernel weight and the wavelength corresponding to each response parameter based on the trained photonic deep convolutional neural network, and store them in the optical weight storage module; the preset iterative strategy is configured to: for each iteration, input image training data into the photonic deep convolutional neural network to obtain the prediction result output by the photonic deep convolutional neural network; determine the loss value based on the prediction result and the ground truth value corresponding to the image training data; for each wavelength in the wavelength combination, determine the corresponding gradient based on the loss value, and update the wavelength combination in reverse; according to each wavelength in the updated wavelength combination, search for the corresponding response parameter again from the optical weight pool, and update the multi-convolutional kernel weights of the photonic deep convolutional neural network; use the updated photonic deep convolutional neural network to execute the next iteration until the maximum number of iterations is reached, to obtain the trained photonic deep convolutional neural network.
[0014] According to the present invention, an image processing system based on a wavelength parameter reconfigurable optical neural network includes a training module, which further includes: a testing unit for testing the optical image processing module configured with a modulation unit and a detection unit to obtain experimental response results; the experimental response results include the response results of each output port; and a calibration unit for obtaining errors based on the wavelength response spectrum of each output port and the experimental response results, and calibrating the optical image processing module based on the errors.
[0015] This invention also provides an image processing method based on a wavelength parameter reconfigurable optical neural network, applied to any of the above-described image processing systems based on a wavelength parameter reconfigurable optical neural network. The method is characterized by: acquiring cached data, wavelength parameters, and a response matrix corresponding to the wavelength parameters; wherein the cached data characterizes the image to be processed as input to the system in its initial state or a previously cached intermediate image processing result; the wavelength parameters and the response matrix corresponding to the wavelength parameters are trained and stored based on a previously constructed photonic deep convolutional neural network; and based on the wavelength parameters and the response matrix corresponding to the wavelength parameters, the cached data is passively modulated to obtain the corresponding probe light intensity, and the probe light intensity is output as the final result or cached as an intermediate image processing result.
[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image processing method based on a wavelength parameter reconstructable optical neural network as described above.
[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method based on a wavelength parameter reconfigurable optical neural network as described above.
[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image processing method based on a wavelength parameter reconfigurable optical neural network as described above.
[0019] The image processing system and method based on wavelength parameter reconfigurable optical neural networks provided by this invention coordinates the data caching module and the optical weight storage module through a synchronization module to coordinate the input of cached data with wavelength parameters and corresponding response parameters. This decouples the network weights from the optical image processing module, thereby reconfiguring the passive diffraction chip's computing function by switching wavelengths without changing the optical path structure. This allows for rapid switching of the network or computing level. Simultaneously, without increasing external hardware computing resources or additional reconstruction algorithms, traditional electronic parameters are transferred to optical wavelength parameters, fully leveraging optical advantages. The optical image processing module passively modulates the optical signal and uses it as cached data for the next iteration or as the final output, constructing a reconfigurable, end-to-end computational closed loop. This achieves all-optical acceleration of complex deep neural network models while maintaining the hardware structure, resulting in a significant improvement in computing speed and energy efficiency. It overcomes the limitations of traditional electronic computing in terms of energy efficiency and bandwidth, and has significant advantages in low power consumption, high parallelism, and light-speed processing. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0021] Figure 1 This is one of the structural schematic diagrams of the image processing system based on a wavelength parameter reconfigurable optical neural network provided by the present invention; Figure 2 This is the second schematic diagram of the image processing system based on a wavelength parameter reconfigurable optical neural network provided by the present invention; Figure 3 This is one of the flowcharts of the image processing method based on wavelength parameter reconfigurable optical neural network provided by the present invention; Figure 4This is the second flowchart of the image processing method based on wavelength parameter reconfigurable optical neural network provided by the present invention; Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0023] Figure 1 This is a flowchart illustrating the image processing system based on a wavelength parameter reconfigurable optical neural network provided by the present invention, as shown below. Figure 1 As shown, the system comprises an optical image processing module, a synchronization module, an optical weight storage module, and a data cache module, wherein: Synchronization module 11 is used to send synchronization signals to the optical weight storage module and the data cache module respectively; Data caching module 12 is used to send cached data to the optical image processing module according to the synchronization signal. The cached data is used to characterize the image to be processed input in the initial state of the system or the intermediate processing result of the image cached by the optical image processing module. The optical weight storage module 13 is used to send the corresponding wavelength parameters and the corresponding response parameters to the optical image processing module according to the synchronization signal. The wavelength parameters and the corresponding response parameters are trained and stored based on the previously constructed photonic deep convolutional neural network. The optical image processing module 14 is used to passively modulate the input buffer data according to the input wavelength parameters and the corresponding response parameters to obtain the corresponding detection light intensity, and output the detection light intensity as the final result, or update the data buffer module as the intermediate image processing result.
[0024] It should be noted that when the system is in its initial state, the cached data is the initial input image to be processed. When the system is not in its initial state, the cached data is the intermediate image processing results cached from the previous system state, specifically determined based on the current computational state of the system. Furthermore, the photonic deep convolutional neural network is first built and trained based on different wavelength parameters and their corresponding response parameters. Through the trained photonic deep convolutional neural network, the final wavelength parameters and their corresponding response parameters are determined. This facilitates programming so that, according to the data stream clock, the optical image processing module is triggered in batches to load the corresponding required wavelength and response parameters. This allows the optical image processing module to utilize the newly loaded wavelength and response parameters, combined with the corresponding input cached data, to perform the corresponding operations.
[0025] Specifically, refer to Figure 2 The optical image processing module includes an encoding unit, a modulation unit, a diffraction core unit, and a detection unit. Specifically: the encoding unit encodes buffered data to obtain an encoded optical signal; the modulation unit modulates the encoded optical signal according to wavelength parameters, modulates the corresponding optical signal according to the response parameters corresponding to the wavelength parameters, and transmits the modulated wavelength division multiplexing optical signal to the diffraction core unit via a waveguide; the diffraction core unit utilizes a subwavelength structure to perform matrix calculations on the wavelength division multiplexing optical signal input through the waveguide to obtain different light field distributions at different spatial locations; and the detection unit detects the light field distribution output by the diffraction core unit to obtain the corresponding detection light intensity.
[0026] It should be added that, regardless of whether the cached data is the image to be processed or the intermediate image processing result, it is an electrical signal. Therefore, before filtering, the cached data needs to be encoded into an optical signal for subsequent processing. The encoding method can be selected according to the actual design requirements, such as a spatial light modulator, etc., without further limitation here.
[0027] In this embodiment, the modulation unit is used to transmit the encoded optical signal through the main waveguide; the modulation unit includes at least one micro-ring modulator, which is used to: couple the optical signal of the corresponding wavelength in the encoded optical signal passing through the coupling region of the micro-ring modulator in the main waveguide to the micro-ring resonant cavity of the micro-ring modulator according to the received wavelength parameters; and perform amplitude modulation on the optical signal of the corresponding wavelength in the micro-ring resonant cavity based on the response parameters corresponding to the wavelength parameters, and release the modulated optical signal into the main waveguide.
[0028] It should be added that the micro-ring modulator needs to pre-control the voltage of the micro-ring resonant cavity according to the response parameters corresponding to the wavelength parameters, thereby controlling its transmittance. This allows for precise modulation of the optical signal amplitude within the resonant region after the micro-ring controls the filtering of the corresponding coded optical signal based on the received wavelength parameters, and the modulated light is then directly released back into the main waveguide. Furthermore, after releasing the modulated optical signal back into the main waveguide, the modulation unit transmits the final wavelength division multiplexed optical signal to the diffraction core unit through a waveguide.
[0029] Furthermore, the modulation unit and the diffraction core unit are led out to the PCB board via high-frequency lines, and each micro-ring modulator has a DC line lead out to facilitate the adjustment of the operating point of the corresponding micro-ring modulator.
[0030] In one alternative embodiment, the number of micro-ring modulators is determined in advance based on the kernel size of the photonic deep convolutional neural network. For example, when the kernel size is 2×2, the number of micro-ring modulators is 4, and no further limitation is made here.
[0031] In addition, when the number of micro-ring modulators is greater than one, each micro-ring modulator receives the corresponding wavelength parameter and the response parameter corresponding to the wavelength parameter, and the micro-ring modulator and the wavelength parameter are in one-to-one correspondence; each micro-ring modulator independently modulates the corresponding optical signal and transmits the obtained wavelength division multiplexing signal to the same waveguide.
[0032] It should be noted that the modulation principle of the micro-ring modulator can be found above, and will not be repeated here.
[0033] In one optional embodiment, the subwavelength structure is configured based on different wavelength transformation methods, which are determined based on the wavelength response spectral lines obtained by scanning the optical image processing module without modulation and detection units. The diffraction core unit is used to project the optical signal of the corresponding wavelength onto the corresponding spatial position of the output plane through the subwavelength structure according to the transformation method of each wavelength.
[0034] It should be noted that, since the wavelength response spectral lines exhibit the complex wavelength modulation effect of the diffraction core unit, the corresponding subwavelength structure is configured according to the wavelength response spectral lines to provide support and guarantee for the subsequent on-chip computation of all-optical reconstruction and programming of the convolution matrix through wavelength switching.
[0035] Furthermore, the diffraction unit uses a subwavelength grating of an optical subwavelength structure to diffract and transmit the input signal, thereby achieving passive optical computing. The diffraction transmission process is that the optical signal is broadcast-propagated and parallel-computed in the passive optical structure through diffraction. This process is equivalent to a programmable end-to-end matrix transformation. Through reverse design, the subwavelength structure can adapt to different wavelengths for transformation, achieving complex modulation. This gives the optical image processing module significant wavelength response characteristics, ensuring that different output results are generated for different input wavelengths. The programmability of wavelength switching avoids the data movement bottleneck in traditional electronic architectures, improving computing efficiency and bandwidth.
[0036] It is worth noting that spatial multiplexing is also required to divide the internal space of the optical image processing module into different channels so as to simultaneously output the independent signals projected onto the corresponding spatial positions of the output plane. The number of channels needs to be determined based on the number of channels of the photonic deep convolutional neural network.
[0037] Accordingly, the detection unit includes a photodetector array. The number of photodetectors in the photodetector array is determined in advance based on the number of channels of the photonic deep convolutional neural network. For example, if the number of channels of the convolution is 8, then the number of photodetectors in the photodetector array is 8. The specific number can be set according to the actual design requirements, and no further limitation is made here.
[0038] In an optional embodiment, the system further includes a training module, specifically comprising: a spectral scanning unit for performing spectral scanning measurements on the optical image processing module without modulation and detection units to obtain the wavelength response spectral lines of each output port; a network construction unit for constructing an all-optical parameterized photonic deep convolutional neural network based on the wavelength response spectral lines of each output port; a training data acquisition unit for acquiring image training data and the ground truth values corresponding to the image training data; and a training unit for training the photonic deep convolutional neural network by using the image training data as input data for training and the ground truth values corresponding to the image training data as labels for training.
[0039] It should be added that when using the scanning unit to perform scanning measurements on an optical image processing module without a modulation unit and a detection unit, the wavelength of the input light is continuously changed, and the optical response of the optical image processing module at each wavelength is measured by a spectrometer to obtain the wavelength response spectrum corresponding to each output end.
[0040] Furthermore, the input light can be generated by a tunable laser, and the polarization state of the laser can be controlled by a polarization controller to ensure that the light input to the optical image processing module has the desired wavelength. Additionally, the chemical response includes optical power, phase, amplitude, transmittance, and loss, which can be selected according to actual design requirements and are not further limited here.
[0041] In addition, the network construction unit is also used to: determine the response parameters corresponding to the wavelength of each channel based on the wavelength response spectrum of each output port, and construct an optical weight pool; the response parameters are used to characterize the transmission characteristics of the corresponding channel, and the channel is used to characterize the transmission path of the wavelength from the input optical image processing module to the output; based on any wavelength combination, find the response parameters of the corresponding wavelength from the optical weight pool and map them to parallel multi-convolution kernel weights to construct a photonic deep convolutional neural network; the number of wavelength parameters in the wavelength combination is configured according to the number of micro-ring modulators in the modulation unit.
[0042] It should be added that the response parameters can be selected from transmittance or loss, etc., and the specific selection can be made according to actual design requirements and prior experience. No further restrictions are made here.
[0043] Furthermore, the training unit is also used to train the photonic deep convolutional neural network according to a preset iterative strategy, and to determine the response parameters corresponding to each convolutional kernel weight and the wavelength corresponding to each response parameter based on the trained photonic deep convolutional neural network, and store them in the optical weight storage module. The preset iterative strategy is used to: input image training data into the photonic deep convolutional neural network for each iteration to obtain the prediction result output by the photonic deep convolutional neural network; determine the loss value based on the prediction result and the ground truth value corresponding to the image training data; determine the corresponding gradient for each wavelength in the wavelength combination based on the loss value, and update the wavelength combination in reverse; search for the corresponding response parameters from the optical weight pool based on each wavelength in the updated wavelength combination, and update the multi-convolutional kernel weights of the photonic deep convolutional neural network; and execute the next iteration using the updated photonic deep convolutional neural network until the maximum number of iterations is reached, thus obtaining the trained photonic deep convolutional neural network.
[0044] It is worth noting that the convergence of the trained photonic deep convolutional neural network can be determined based on the loss curve obtained during the iterative training process.
[0045] In an optional embodiment, the training module further includes: a testing unit for testing the optical image processing module configured with a modulation unit and a detection unit to obtain experimental response results; the experimental response results include the response results of each output port; and a calibration unit for obtaining errors based on the wavelength response spectrum of each output port and the experimental response results, and calibrating the optical image processing module based on the errors.
[0046] It should be noted that when testing the optical image processing module, a dynamic test signal is generated by an arbitrary waveform generator to simulate real working data, and a static and stable bias voltage is provided by a DC source to configure and calibrate the operating point of the optical image processing module. An oscilloscope is used to capture the dynamic electrical signal output by the optical image processing module to obtain the time delay response result, which is then used for subsequent result analysis and performance evaluation with the wavelength response spectrum of each output port.
[0047] Furthermore, the error results between the pure optical acquisition of the spectral scanning unit and the high-speed modulation detection of the testing unit are used as a perturbation to the network wavelength parameters, making the entire system more complete and robust. This fusion optimization algorithm based on the above offline training and online calibration not only ensures the consistency between the online training algorithm and the actual experimental results, but also retains the training time and cost advantages of offline training in building deep neural networks. It is compatible with the current mainstream optimization schemes, greatly improves the training speed of the network, and provides a strong guarantee for the all-optical programmable deep neural network architecture.
[0048] In an alternative embodiment, based on the above system, software collaboration can be used to validate its performance and advantages on real-world image tasks. For example, for an optical VGG16 convolutional neural network, a complete VGG16 network with optically mapped convolutional parameters is constructed by selecting multiple optical wavelength groups to map multi-kernel, multi-channel convolutional operations. This network is then used for classification tasks on the CIFAR10 and CIFAR100 datasets. After error calibration, performance degradation is not significant, and it still achieves accuracy almost equivalent to that of electronic networks.
[0049] In addition, for optical image style transfer networks, using the same construction method as the above multi-core multi-channel convolution, multiple optical image style convolution libraries are constructed by selecting multiple wavelength combinations, and the image feature maps are processed in the encoded latent space. After decoding, different stylized output results are obtained.
[0050] In summary, this invention coordinates the data caching module and the optical weight storage module through a synchronization module to coordinate the input of cached data with wavelength parameters and corresponding response parameters. This decouples the network weights from the optical image processing module, thereby reconstructing the passive diffraction chip's computational function through wavelength switching without altering the optical path structure. This allows for rapid switching of network or computational levels. Simultaneously, without increasing external hardware computing resources or additional reconstruction algorithms, traditional electronic parameters are transferred to optical wavelength parameters, fully leveraging optical advantages. The optical image processing module passively modulates the optical signal and uses it as cached data for the next iteration or as the final output, constructing a reconstructable, end-to-end computational closed loop. This achieves all-optical acceleration of complex deep neural network models while maintaining the hardware structure, resulting in a significant improvement in computational speed and energy efficiency. It overcomes the limitations of traditional electronic computing in terms of energy efficiency and bandwidth, offering significant advantages such as low power consumption, high parallelism, and light-speed processing.
[0051] The image processing method based on wavelength parameter reconfigurable optical neural network provided by the present invention is described below. The image processing method based on wavelength parameter reconfigurable optical neural network described below can be referred to in correspondence with the image processing system based on wavelength parameter reconfigurable optical neural network described above.
[0052] Figure 3 A flowchart illustrating an image processing method based on a wavelength parameter reconfigurable optical neural network is shown, applicable to any of the image processing systems based on wavelength parameter reconfigurable optical neural networks described above. The method includes: S31, obtain cached data, wavelength parameters, and the response matrix corresponding to the wavelength parameters; wherein, the cached data is used to characterize the image to be processed in the initial state input of the system or the intermediate processing result of the previously cached image, and the wavelength parameters and the response matrix corresponding to the wavelength parameters are trained and stored based on the previously constructed photonic deep convolutional neural network; S32, based on the wavelength parameters and the response matrix corresponding to the wavelength parameters, performs passive modulation on the buffered data to obtain the corresponding probe light intensity, and outputs the probe light intensity as the final result, or as a buffer for intermediate image processing results.
[0053] It should be noted that the step number "S3N" in this specification does not represent the order of the image processing method based on wavelength parameter reconfigurable optical neural network. The image processing method based on wavelength parameter reconfigurable optical neural network of the present invention is described in detail below.
[0054] Step S31: Obtain cached data, wavelength parameters, and the response matrix corresponding to the wavelength parameters; wherein, the cached data is used to characterize the image to be processed in the initial state input of the system or the intermediate processing result of the previously cached image, and the wavelength parameters and the response matrix corresponding to the wavelength parameters are trained and stored based on the previously constructed photonic deep convolutional neural network; Step S32: Based on the wavelength parameters and the response matrix corresponding to the wavelength parameters, passively modulate the buffered data to obtain the corresponding probe light intensity, and output the probe light intensity as the final result, or as the buffer of the intermediate image processing result.
[0055] In this embodiment, the buffered data is passively modulated according to the wavelength parameters and the response matrix corresponding to the wavelength parameters to obtain the corresponding probe light intensity. This includes: encoding the buffered data to obtain an encoded optical signal; modulating the encoded optical signal according to the wavelength parameters; modulating the corresponding optical signal according to the response parameters corresponding to the wavelength parameters; using a subwavelength structure, performing matrix calculations on the wavelength division multiplexed optical signal input through the waveguide to obtain different light field distributions at different spatial locations; and probing the light field distribution output by the diffraction core unit to obtain the corresponding probe light intensity.
[0056] Specifically, the encoded optical signal is filtered according to the wavelength parameter, and the filtered optical signal is modulated according to the response parameter corresponding to the wavelength parameter. This includes: coupling the optical signal of the corresponding wavelength in the encoded optical signal passing through the coupling region in the main waveguide to the micro-ring resonant cavity according to the received wavelength parameter; performing amplitude modulation on the optical signal of the corresponding wavelength in the micro-ring resonant cavity based on the response parameter corresponding to the wavelength parameter; and releasing the modulated optical signal into the main waveguide.
[0057] In addition, before projecting the wavelength division multiplexing optical signal input through the waveguide to the corresponding spatial position on the output plane using the subwavelength structure, the process includes: configuring the transformation mode of the subwavelength structure for different wavelengths based on the wavelength response spectrum obtained by scanning the spectrum of the optical image processing module without modulation and detection units.
[0058] In one optional embodiment, before passively modulating the buffered data according to the wavelength parameters and the response matrix corresponding to the wavelength parameters, the method includes: performing a spectral sweep measurement on the optical image processing module without modulation and detection units to obtain the wavelength response spectrum of each output port; constructing an all-optical parameterized photonic deep convolutional neural network according to the wavelength response spectrum of each output port; obtaining image training data and the ground truth value corresponding to the image training data; using the image training data as input data for training and the ground truth value corresponding to the image training data as a label for training, and training the photonic deep convolutional neural network.
[0059] It should be added that, for reference Figure 3When performing a spectrum sweep measurement on an optical image processing module without a modulation unit and a detection unit, the wavelength of the input light is continuously changed, and the optical response of the optical image processing module at each wavelength is measured by a spectrometer to obtain the wavelength response spectrum corresponding to each output end.
[0060] Furthermore, based on the wavelength response spectrum of each output port, a fully optically parameterized photonic deep convolutional neural network is constructed, including: determining the response parameters corresponding to the wavelength of each channel based on the wavelength response spectrum of each output port, and constructing an optical weight pool; the response parameters are used to characterize the transmission characteristics of the corresponding channel, and the channel is used to characterize the transmission path of the wavelength from the input optical image processing module to the output; based on any wavelength combination, the response parameters of the corresponding wavelength are found from the optical weight pool and mapped to parallel multi-convolutional kernel weights to construct the photonic deep convolutional neural network; the number of wavelength parameters in the wavelength combination is configured according to the number of micro-ring modulators in the modulation unit.
[0061] In addition, image training data is used as input data for training, and the ground truth values corresponding to the image training data are used as labels for training. The photonic deep convolutional neural network is trained using: image training data as input data and the ground truth values corresponding to the image training data as labels, following a preset iterative strategy. Based on the trained photonic deep convolutional neural network, the response parameters corresponding to each convolutional kernel weight and the wavelengths corresponding to each response parameter are determined and stored in the optical weight storage module. The preset iterative strategy is used to: for each iteration, the image training data... The input data is fed into the photonic deep convolutional neural network to obtain the predicted results output by the photonic deep convolutional neural network. Based on the predicted results and the ground truth values corresponding to the image training data, the loss value is determined. For each wavelength in the wavelength combination, the corresponding gradient is determined separately in combination with the loss value, and the wavelength combination is updated in reverse. Based on each wavelength in the updated wavelength combination, the corresponding response parameters are searched again from the optical weight pool, and the multi-convolution kernel weights of the photonic deep convolutional neural network are updated. The next iteration is performed using the updated photonic deep convolutional neural network until the maximum number of iterations is reached, resulting in the trained photonic deep convolutional neural network.
[0062] In an alternative embodiment, continue to refer to Figure 3 Before passively modulating the buffered data based on the wavelength parameters and the corresponding response matrix, the process also includes: testing the optical image processing module equipped with modulation and detection units to obtain experimental response results; the experimental response results include the response results of each output port; and obtaining the error based on the wavelength response spectrum of each output port and the experimental response results, and calibrating the optical image processing module based on the error.
[0063] In summary, this invention decouples network weights from the optical image processing module by coordinating the input of cached data, wavelength parameters, and corresponding response parameters. This allows for the reconstruction of the passive diffraction chip's computational function through wavelength switching without altering the optical path structure. This enables rapid switching of network or computational levels. Simultaneously, without increasing external hardware computing resources or requiring additional reconstruction algorithms, traditional electronic parameters are transferred to optical wavelength parameters, fully leveraging optical advantages to passively modulate the optical signal. This signal is then used as cached data for the next iteration or as the final output, constructing a reconfigurable, end-to-end computational closed loop. This achieves all-optical acceleration of complex deep neural network models while maintaining the hardware structure, resulting in a significant improvement in computational speed and energy efficiency. It overcomes the limitations of traditional electronic computing in terms of energy efficiency and bandwidth, offering significant advantages such as low power consumption, high parallelism, and light-speed processing.
[0064] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an image processing method based on a wavelength parameter reconfigurable optical neural network. This method includes: acquiring cached data, wavelength parameters, and a response matrix corresponding to the wavelength parameters; wherein the cached data is used to characterize the image to be processed as the initial input of the system or a previously cached intermediate image processing result; the wavelength parameters and the response matrix corresponding to the wavelength parameters are trained and stored based on a previously constructed photonic deep convolutional neural network; passively modulating the cached data according to the wavelength parameters and the response matrix corresponding to the wavelength parameters to obtain the corresponding probe light intensity, and outputting the probe light intensity as the final result or as a cached intermediate image processing result.
[0065] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0066] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image processing method based on a wavelength parameter reconfigurable optical neural network provided by the above methods. The method includes: acquiring cached data, wavelength parameters, and a response matrix corresponding to the wavelength parameters; wherein the cached data is used to characterize the image to be processed input in the initial state of the system or the intermediate processing result of the previously cached image, and the wavelength parameters and the response matrix corresponding to the wavelength parameters are trained and stored based on a previously constructed photonic deep convolutional neural network; passively modulating the cached data according to the wavelength parameters and the response matrix corresponding to the wavelength parameters to obtain the corresponding detection light intensity, and outputting the detection light intensity as the final result or as a cached intermediate processing result of the image.
[0067] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an image processing method based on a wavelength parameter reconfigurable optical neural network provided by the methods described above. The method includes: acquiring cached data, wavelength parameters, and a response matrix corresponding to the wavelength parameters; wherein the cached data is used to characterize the image to be processed input to the initial state of the system or a previously cached intermediate image processing result, and the wavelength parameters and the response matrix corresponding to the wavelength parameters are trained and stored based on a previously constructed photonic deep convolutional neural network; passively modulating the cached data according to the wavelength parameters and the response matrix corresponding to the wavelength parameters to obtain the corresponding probe light intensity, and outputting the probe light intensity as the final result or as a cached intermediate image processing result.
[0068] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0069] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An image processing system based on wavelength parameter reconfigurable optical neural network, characterized in that, It includes an optical image processing module, a synchronization module, an optical weight storage module, and a data caching module, among which: The synchronization module is used to send synchronization signals to the optical weight storage module and the data cache module respectively. The data caching module is used to send cached data to the optical image processing module according to the synchronization signal. The cached data is used to characterize the image to be processed input in the initial state of the system or the intermediate processing result of the image cached by the optical image processing module. The optical weight storage module is used to send the corresponding wavelength parameters and the corresponding response parameters to the optical image processing module according to the synchronization signal. The wavelength parameters and the corresponding response parameters are trained and stored based on a previously constructed photonic deep convolutional neural network. The optical image processing module is used to passively modulate the input cached data according to the input wavelength parameters and the response parameters corresponding to the wavelength parameters to obtain the corresponding detection light intensity, and output the detection light intensity as the final result, or update the data cache module as the intermediate image processing result.
2. The wavelength parameter reconfigurable optical neural network based image processing system of claim 1, wherein, The optical image processing module includes an encoding unit, a modulation unit, a diffraction core unit, and a detection unit, wherein: The encoding unit is used to encode the cached data to obtain an encoded optical signal; The modulation unit is used to modulate the coded optical signal according to the wavelength parameter, modulate the corresponding optical signal according to the response parameter corresponding to the wavelength parameter, and transmit the modulated wavelength division multiplexed optical signal to the diffraction core unit through a waveguide. The diffraction core unit is used to perform matrix calculations on the wavelength division multiplexed optical signal input through the waveguide using a subwavelength structure to obtain different optical field distributions in spatial locations. The detection unit is used to detect the light field distribution output by the diffraction core unit and obtain the corresponding detection light intensity.
3. The image processing system based on a wavelength parameter reconfigurable optical neural network according to claim 2, characterized in that, The modulation unit is used to transmit the encoded optical signal through the main waveguide; the modulation unit includes at least one micro-ring modulator, the micro-ring modulator being used for: Based on the received wavelength parameters, the optical signal of the corresponding wavelength in the encoded optical signal that passes through the coupling region of the micro-ring modulator in the main waveguide is coupled to the micro-ring resonant cavity of the micro-ring modulator. Based on the response parameters corresponding to the wavelength parameters, the optical signal of the corresponding wavelength in the micro-ring resonant cavity is amplitude modulated, and the modulated optical signal is released into the main waveguide.
4. The image processing system based on a wavelength parameter reconfigurable optical neural network according to claim 3, characterized in that, The number of micro-ring modulators is determined in advance based on the kernel size of the photonic deep convolutional neural network; When the number of micro-ring modulators is greater than one, for each micro-ring modulator, the corresponding wavelength parameter and the response parameter corresponding to the wavelength parameter are received respectively, and the micro-ring modulator and the wavelength parameter are in one-to-one correspondence. For each of the micro-ring modulators, the corresponding optical signal is modulated independently, and the resulting wavelength division multiplexed signal is transmitted to the same waveguide.
5. The image processing system based on a wavelength parameter reconfigurable optical neural network according to claim 2, characterized in that, The subwavelength structure is configured based on different wavelength transformation methods. The different wavelength transformation methods are determined based on the wavelength response spectrum obtained by scanning the spectrum of an optical image processing module without the modulation unit and the detection unit. The diffraction core unit is used to project the light signal of the corresponding wavelength onto the corresponding spatial position of the output plane through the subwavelength structure according to the wavelength transformation method. The detection unit includes a photodetector array, the number of which is determined in advance based on the number of channels of the photonic deep convolutional neural network.
6. The image processing system based on a wavelength parameter reconfigurable optical neural network according to claim 2, characterized in that, It also includes a training module, specifically including: The spectral scanning unit is used to perform spectral scanning measurements on the optical image processing module that is not configured with the modulation unit and the detection unit to obtain the wavelength response spectral lines of each output port; The network construction unit is used to construct a fully optical parameterized photonic deep convolutional neural network based on the wavelength response spectral lines of each output port. The training data acquisition unit is used to acquire image training data and the ground values corresponding to the image training data; The training unit is used to train the photonic deep convolutional neural network by using the image training data as input data for training and the ground truth values corresponding to the image training data as labels for training.
7. The image processing system based on a wavelength parameter reconfigurable optical neural network according to claim 6, characterized in that, The network construction unit is also used for: Based on the wavelength response spectrum of each output port, the response parameters corresponding to the wavelength of each channel are determined, and an optical weight pool is constructed; the response parameters are used to characterize the transmission characteristics of the corresponding channel, and the channel is used to characterize the transmission path of the wavelength from the input of the optical image processing module to the output; Based on any wavelength combination, the response parameters corresponding to the wavelength are found from the optical weight pool and mapped to parallel multi-convolutional kernel weights to construct a photonic deep convolutional neural network; the number of wavelength parameters in the wavelength combination is configured according to the number of micro-ring modulators in the modulation unit.
8. The image processing system based on a wavelength parameter reconfigurable optical neural network according to claim 7, characterized in that, The training unit is further configured to train the photonic deep convolutional neural network according to a preset iteration strategy, and determine the response parameters corresponding to each convolutional kernel weight and the wavelength corresponding to each response parameter based on the trained photonic deep convolutional neural network, and store them in the optical weight storage module; the preset iteration strategy is used for: For each iteration, the image training data is input into the photonic deep convolutional neural network to obtain the prediction result output by the photonic deep convolutional neural network; The loss value is determined based on the prediction result and the ground truth value corresponding to the image training data; For each wavelength in the wavelength combination, the corresponding gradient is determined based on the loss value, and the wavelength combination is updated in reverse. Based on each wavelength in the updated wavelength combination, the corresponding response parameters are re-searched from the optical weight pool, and the multi-convolution kernel weights of the photonic deep convolutional neural network are updated. The next iteration is performed using the updated photonic deep convolutional neural network until the maximum number of iterations is reached, resulting in a trained photonic deep convolutional neural network.
9. The image processing system based on a wavelength parameter reconfigurable optical neural network according to claim 6, characterized in that, The training module also includes: The test unit is used to test the optical image processing module configured with the modulation unit and the detection unit to obtain experimental response results; the experimental response results include the response results of each output port. The calibration unit is used to obtain the error based on the wavelength response spectrum of each output port and the experimental response results, and to calibrate the optical image processing module based on the error.
10. An image processing method based on a wavelength parameter reconfigurable optical neural network, applied to an image processing system based on a wavelength parameter reconfigurable optical neural network as described in any one of claims 1-9, characterized in that, include: Obtain cached data, wavelength parameters, and the response matrix corresponding to the wavelength parameters; wherein, the cached data is used to characterize the image to be processed input to the initial state of the system or the intermediate processing result of the previously cached image, and the wavelength parameters and the response matrix corresponding to the wavelength parameters are trained and stored based on a previously constructed photonic deep convolutional neural network; Based on the wavelength parameter and the response matrix corresponding to the wavelength parameter, the cached data is passively modulated to obtain the corresponding probe light intensity, and the probe light intensity is output as the final result or cached as the intermediate image processing result.