Optoelectronic hybrid computing system and method and apparatus for training the same

Abstract

The present disclosure provides an optoelectronic hybrid computing system and a training method and device thereof, and relates to the fields of optical computing, metasurface devices, computational imaging and artificial intelligence. The system can comprise: an optical encoder, a light splitting module, a spatial domain light path branch, a frequency domain light path branch and a target neural network; the optical encoder is configured to perform optical encoding on an input light field to obtain an encoded light field; the light splitting module is configured to split the encoded light field into a first light beam and a second light beam; the spatial domain light path branch is configured to generate a spatial domain encoded image according to the first light beam; the frequency domain light path branch is configured to generate a frequency domain encoded image according to the second light beam; and the target neural network is configured to generate a task processing result corresponding to the input light field according to the spatial domain encoded image and the frequency domain encoded image. By applying the scheme of the present disclosure, the computational complexity and power consumption of the target neural network can be effectively reduced, and the task processing efficiency can be improved.

Description

Technical Field

[0001] This disclosure relates to fields such as optical computing, metasurface devices, computational imaging, and artificial intelligence, and particularly to optoelectronic hybrid computing systems and their training methods and apparatus. Background Technology

[0002] Currently, image processing and computer vision systems typically employ convolutional neural networks to process the spatial domain pixels of an image layer by layer. The object of this processing can be an input image obtained by a photodetector recording the intensity of the input light field, and the task processing result corresponding to the input image can be generated based on the layer-by-layer processing results.

[0003] Convolutional neural networks (CNNs) possess characteristics such as local connectivity, weight sharing, and spatial translation equivalence, enabling them to effectively extract edge, texture, and local structural information from images. However, for spatial domain CNNs to extract global periodic structures, frequency components, cross-regional correlations, and long-distance dependencies in images, it is often necessary to stack deeper networks, enlarge convolutional kernels, or introduce attention modules, thereby increasing computational cost and power consumption. Summary of the Invention

[0004] This disclosure provides a hybrid optoelectronic computing system and its training method and apparatus.

[0005] A hybrid optoelectronic computing system includes: an optical encoder, a beam splitting module, a spatial domain optical path branch, a frequency domain optical path branch, and a target neural network; The optical encoder is used to optically encode the input light field to obtain the encoded light field; The beam splitting module is used to split the coded light field into a first beam and a second beam; The spatial domain optical path branch is used to generate a spatial domain encoded image based on the first beam. The frequency domain optical path branch is used to generate a frequency domain encoded image based on the second beam. The target neural network is used to generate the task processing result corresponding to the input light field based on the spatial domain encoded image and the frequency domain encoded image.

[0006] A training method for a hybrid optoelectronic computing system includes: Obtain training data; The optoelectronic hybrid computing system is trained based on the training data, and the optoelectronic hybrid computing system is the system described above.

[0007] A training device for a hybrid optoelectronic computing system includes: a data acquisition module and a system training module; The data acquisition module is used to acquire training data; The system training module is used to train the optoelectronic hybrid computing system based on the training data, and the optoelectronic hybrid computing system is the system described above.

[0008] An electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0009] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram of the composition structure of the first embodiment 100 of the optoelectronic hybrid computing system described in this disclosure; Figure 2 This is a schematic diagram of the composition structure of the second embodiment 200 of the optoelectronic hybrid computing system described in this disclosure; Figure 3 This is a schematic diagram of the composition structure of the frequency domain optical path branch 104 described in this disclosure; Figure 4 This is a schematic diagram of the composition structure of the target neural network 105 described in this disclosure; Figure 5 This is a flowchart illustrating an embodiment of the training method for the optoelectronic hybrid computing system described in this disclosure; Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0014] Figure 1 This is a schematic diagram of the structural composition of the first embodiment 100 of the optoelectronic hybrid computing system described in this disclosure. Figure 1 As shown, it includes: an optical encoder 101, a beam splitting module 102, a spatial domain optical path branch 103, a frequency domain optical path branch 104, and a target neural network 105.

[0015] The optical encoder 101 is used to optically encode the input light field to obtain the encoded light field.

[0016] The beam splitting module 102 is used to split the coded light field into a first beam and a second beam.

[0017] Spatial domain optical path branch 103 is used to generate a spatial domain encoded image based on the first beam.

[0018] Frequency domain optical path branch 104 is used to generate a frequency domain encoded image based on the second beam.

[0019] The target neural network 105 is used to generate the task processing result corresponding to the input light field based on the spatial domain encoded image and the frequency domain encoded image.

[0020] By adopting the scheme described in the above system embodiment, the input light field is optically encoded directly by the optical encoder 101 before photoelectric conversion, which can preserve and enhance task-related features. Then, the encoded light field is split into two beams by the beam splitting module 102. One beam is generated through the spatial domain optical path branch 103 to generate a spatial domain encoded image that maintains the spatial position relationship to extract local structural features. The other beam is generated through the frequency domain optical path branch 104 to generate a frequency domain encoded image to extract global periodicity, frequency and long-distance dependence features. Finally, the target neural network 105 performs joint processing on the spatial domain and frequency domain encoded images, thereby achieving synchronous acquisition and fusion of complementary information in both domains without stacking deep networks or expanding convolutional kernels. This effectively reduces the computational load and power consumption of the target neural network and improves task processing efficiency.

[0021] The following are respectively for Figure 1 Each component is described in detail.

[0022] 1) Optical encoder 101 In some embodiments of this disclosure, the optical encoder 101 may include a metasurface optical encoder, which may employ a static metasurface and / or a tunable metasurface.

[0023] Tunable methods may include: liquid crystal tuning, phase change material tuning, electro-optic tuning, thermo-optic tuning, mechanical tuning, microelectromechanical tuning, and carrier injection tuning.

[0024] Metasurface optical encoders can perform the following functions: phase encoding, amplitude encoding, polarization encoding, wavelength multiplexing, spatial mode multiplexing, edge enhancement, orientation selection, multi-scale filtering, feature compression, optical convolution, random projection of light field, and trainable optical encoding for specific neural network tasks.

[0025] By employing a metasurface optical encoder as the optical encoder 101, passive, low-static-power optical encoding can be achieved using a static metasurface. The encoding parameters can also be dynamically adjusted through various tunable methods, enabling flexible encoding of the input light field in multiple ways, such as amplitude, phase, polarization, wavelength, and spatial mode. This allows for the extraction and enhancement of task-related features of the input light field before photoelectric conversion, providing richer and more suitable encoded light field information for subsequent parallel sensing in both the spatial and frequency domains.

[0026] The optical encoder 101 can be composed of one or more metasurfaces. Each metasurface can be composed of several subwavelength structural units (such as nanopillars, nanopores, or nanofins). The structural parameters of each subwavelength structural unit can include width (w), length (l), height (h), period (p), and in-plane rotation angle (θ). The structural parameters are optimized variables and can be jointly trained with the network parameters of the target neural network 105 through a backpropagation algorithm. In addition, the subwavelength structural units can be made of transparent dielectric materials such as amorphous silicon (a-Si), titanium dioxide (TiO2), and silicon nitride (Si3N4).

[0027] Assuming the input light field is E0(x,y) and the complex modulation function of the optical encoder 101 is Henc(x,y), then the obtained coded light field can be: Ee(x,y)=Henc(x,y)E0(x,y); (1) When the optical encoder 101 is composed of multiple metasurfaces, the encoded light field can be expressed as: Ee(x,y)=PnHnPn-1Hn-1...P1H1E0(x,y); (2) Where Ee(x,y) represents the encoded optical field, H represents the complex modulation function of each metasurface, P represents the free space propagation operator between adjacent metasurfaces, and n represents the number of metasurfaces.

[0028] The optical encoder 101 is positioned before the beam splitting module 102, so the spatial domain optical path branch 103 and the frequency domain optical path branch 104 share the same optical encoding result.

[0029] 2) Spectrometer 102 The encoded optical field can be split into a first beam and a second beam by the beam splitting module 102. The first beam enters the spatial domain optical path branch 103, and the second beam enters the frequency domain optical path branch 104.

[0030] The beam splitting module 102 can be: a beam splitting prism, a semi-transparent and semi-reflective mirror, a polarization beam splitter, a diffraction beam splitter, a metasurface beam splitter, a wavelength beam splitter, or an integrated optical beam splitter.

[0031] In addition, the splitting ratio can be a fixed ratio or it can be flexibly set according to the optical power requirements of the two branches.

[0032] Assuming the power ratio of the spatial domain optical path branch 103 is ηs and the power ratio of the frequency domain optical path branch 104 is ηf, then: ηs+ηf≤1;(3) When the sum of the two is less than 1, the remaining part can correspond to device loss and propagation loss.

[0033] 3) Spatial domain optical path branch 103 and frequency domain optical path branch 104 Figure 2 This is a schematic diagram of the structural composition of the second embodiment 200 of the optoelectronic hybrid computing system described in this disclosure. Figure 2 As shown, the spatial domain optical path branch 103 may include a spatial domain optical module 1031 and a first photodetector 1032, and the frequency domain optical path branch 104 may include a frequency domain optical module 1041 and a second photodetector 1042.

[0034] The two branches will be explained in detail below.

[0035] 3.1) Spatial Domain Optical Path Branch 103 In some embodiments of this disclosure, the spatial domain optical module 1031 can be used to perform spatial domain optical processing on the first beam to obtain a first processing result, and the first photodetector 1032 can be used to generate a spatial domain coded image based on the first processing result.

[0036] That is, the spatial domain optical path branch 103 can maintain the spatial positional relationship of the encoded optical field and complete the photoelectric conversion on the spatial image plane to form a spatial domain encoded image.

[0037] The spatial domain optical module 1031 may include: a free space propagation region, an imaging lens, a metasurface lens, a relay imaging system, a scaling imaging system, a distortion correction device, and an optical filter.

[0038] The light intensity Is(x,y) formed by the spatial domain optical module 1031 on the first photodetector 1032 can be: Is(x,y)=∣I{Ee(x,y)}∣ 2 (4) Under ideal unit magnification imaging conditions, we have: Is(x,y)≈∣Ee(x,y)∣ 2 (5) Where I represents the imaging propagation operator of the spatial domain optical module 1031.

[0039] The first photodetector 1032 may include: a complementary metal-oxide-semiconductor (CMOS) image sensor, a charge-coupled device (CCD) image sensor, a photodiode array, an avalanche photodiode array, a single-photon detection array, an event camera, or other two-dimensional photodetectors.

[0040] 3.2) Frequency domain optical path branch 104 The frequency domain optical path branch 104 may include a frequency domain optical module 1041 and a second photodetector 1042.

[0041] In some embodiments of this disclosure, the frequency domain optical module 1041 can be used to perform frequency domain optical processing on the second beam to obtain a second processing result, and the second photodetector 1042 can be used to generate a frequency domain coded image based on the second processing result.

[0042] The frequency domain optical module 1041 may include at least one Fourier transform optical element, and the second photodetector 1042 may be disposed near its rear focal plane. The Fourier transform optical element may include: a conventional refractive lens, a Fourier transform metasurface lens, a diffractive optical element, a reflective Fourier transform element, or an integrated optical Fourier transform.

[0043] When the focal length of the Fourier transform optical element is f and the incident light wavelength is λ, the light field at the rear focal plane is proportional to the Fourier transform of the encoded light field. The spatial frequency coordinates and the spatial coordinates of the Fourier plane satisfy the following relationship: the spatial frequency in the x-direction is equal to the x-direction offset coordinate relative to the zero-frequency center on the Fourier plane divided by the product of the wavelength and the focal length; the same applies to the y-direction. The light intensity obtained by the second photodetector 1042 in the Fourier plane is the modulus square of the Fourier transform of the encoded light field.

[0044] It can be seen that the frequency domain optical module 1041 in the scheme described in this disclosure does not need to constitute a complete 4-focal-length (4f) imaging system. Instead, it can adopt the Fourier transform part of a complete 4f imaging system, namely: encoded light field → Fourier transform optical element → second photodetector 1042 (back focal plane of lens).

[0045] In some embodiments of this disclosure, the frequency domain optical path branch 104 may further include a frequency domain modulator 1043, and correspondingly, Figure 3 This is a schematic diagram of the composition structure of the frequency domain optical path branch 104 described in this disclosure.

[0046] The frequency domain modulator 1043 can be used to perform frequency domain modulation on the second processing result to obtain the modulation result. Correspondingly, the second photodetector 1042 can generate a frequency domain encoded image based on the modulation result.

[0047] The frequency domain modulator 1043 can be: a metasurface frequency filter, a spatial light modulator, a digital micromirror device, a fixed aperture, an adjustable aperture, a ring filter, a direction-selective filter, or a multi-band filter.

[0048] The frequency domain modulator 1043 has a corresponding transfer function. After the encoded light field undergoes a Fourier transform to obtain the spectrum (second processing result), it is then modulated by the frequency domain modulator 1043. Finally, the second photodetector 1042 collects the modulated light intensity information. By configuring the transmittance, phase, or polarization response of the frequency domain modulator 1043 in different frequency regions, image information corresponding to specific directions, scales, or frequency bands can be enhanced in a targeted manner.

[0049] Correspondingly, by using the frequency domain modulator 1043 to modulate the spectrum at the optical level, the frequency domain features required for the task can be directly screened and enhanced. Frequency domain information pre-enhancement is completed before photoelectric conversion, enriching the effective features of the frequency domain encoded image, complementing the spatial domain information, and reducing the computational pressure on the back-end neural network.

[0050] Furthermore, ordinary photodetectors (such as the second photodetector 1042) can typically only directly measure light intensity, and the frequency domain optical module 1041 mainly obtains spectral intensity information, without directly containing complete spectral phase information. To retain richer phase correlation features in the frequency domain encoded image, the scheme described in this disclosure can employ one or more of the following processing methods: using an optical encoder 101 (such as a metasurface optical encoder) for phase-to-intensity mapping; setting multiple metasurface encoding channels with different phase responses; employing multiple exposures or phase diversity measurements; introducing reference light to form interferometry; using polarization multiplexing to obtain multiple phase correlation channels; using wavelength multiplexing to obtain spectral intensity under different propagation phases; and setting an off-axis holographic structure to recover the complex spectrum.

[0051] Taking the recovery of complex spectrum by setting an off-axis holographic structure as an example, in some embodiments of this disclosure, the second photodetector 1042 can generate an interference fringe image based on the interference result of the reference beam and the second processing result. The reference beam can be an inclined beam set in front of the frequency domain optical module 1041. Correspondingly, the frequency domain optical path branch 104 may also include a post-processing unit (not shown for the sake of simplifying the drawings) for generating a frequency domain encoded image based on the interference fringe image.

[0052] The angle between the reference beam and the principal optical axis can be determined according to actual needs, and is usually a small angle. The reference beam will interfere with the second processing result on the surface of the second photodetector 1042, generating an interference fringe image. If... Figure 3 As shown, the frequency domain optical path branch 104 includes a frequency domain modulator 1043. The reference beam will then interfere with the modulation result on the surface of the second photodetector 1042, generating an interference fringe image. Subsequently, the post-processing unit can spatially filter the interference fringe image, extract the carrier term, and perform an inverse Fourier transform to obtain the amplitude and phase of the complex spectrum. Based on the amplitude and phase, the frequency domain coded image is determined.

[0053] By adopting the above processing method, the problem that ordinary photodetectors can only measure light intensity and lose phase information is solved, so that the frequency domain encoded image can contain more complete frequency domain features, which enhances the ability of the subsequent target neural network 105 to perceive tasks such as texture, edge and fine structure.

[0054] In the scheme described in this disclosure, the first photodetector 1032 and the second photodetector 1042 can achieve synchronous or quasi-synchronous exposure through a synchronous acquisition module to ensure that the spatial domain coded image and the frequency domain coded image originate from the same input scene.

[0055] The synchronous acquisition module may include: a unified hardware trigger, a field-programmable gate array (FPGA) controller, a synchronous clock, an exposure control module, a light source pulse control module, and a CMOS horizontal and vertical synchronization module.

[0056] Furthermore, since the pixel size, resolution, optical axis position, and scaling ratio of the first photodetector 1032 and the second photodetector 1042 may differ, calibration can also be performed. Calibration parameters may include: geometric distortion of the spatial domain optical module 1031, Fourier plane center position, zero-frequency position, frequency coordinate scale, rotation and flipping relationship of the first photodetector 1032 and the second photodetector 1042, light intensity response non-uniformity, exposure time difference, light power ratio, dark current and noise, wavelength, focal length, and pixel size, etc.

[0057] Furthermore, before inputting the spatial domain encoded image and the frequency domain encoded image into the target neural network 105, the two images can be size-adapted. For example, Xf can be placed to the same spatial size as Xs through bilinear interpolation or adaptive average pooling, or they can be input into branch networks with different downsampling rates, and then fused at the feature level after adjusting the number of channels through 1×1 convolution. Here, Xf represents the frequency domain encoded image, and Xs represents the spatial domain encoded image.

[0058] 4) Target Neural Network 105 In some embodiments of this disclosure, the target neural network 105 may include: a spatial domain electronic branch 1051, a frequency domain electronic branch 1052, and a cross-domain fusion unit 1053. Accordingly, Figure 4 This is a schematic diagram of the composition structure of the target neural network 105 described in this disclosure.

[0059] The spatial domain electronic branch 1051 can be used to extract spatial domain features from the spatial domain encoded image, the frequency domain electronic branch 1052 can be used to extract frequency domain features from the frequency domain encoded image, and the cross-domain fusion unit 1053 can be used to fuse spatial domain features and frequency domain features, and generate task processing results based on the fusion results.

[0060] The spatial domain electron branch 1051 can be represented as: Fs=Gs(Xs;θs); (6) Frequency domain electronic branch 1052 can be represented as: Ff=Gf(Xf;θf); (7) Where Fs represents the spatial domain features output by the spatial domain electronic branch 1051, Gs represents the spatial domain feature extraction network, θs represents the network parameters of the spatial domain feature extraction network, Ff represents the frequency domain features output by the frequency domain electronic branch 1052, Gf represents the frequency domain feature extraction network, and θf represents the network parameters of the frequency domain feature extraction network.

[0061] Spatial domain feature extraction networks may include: convolutional neural networks, residual networks, lightweight convolutional networks, vision transformers, state-space models, and hybrid convolutional and attention networks.

[0062] Frequency domain feature extraction networks may include: frequency-band convolutional networks, polar coordinate spectrum networks, ring frequency band networks, directional frequency coding networks, multi-scale frequency networks, spectrum Transformers, complex neural networks, and neural networks based on radial and angular frequencies.

[0063] By employing the above processing method, spatial domain features and frequency domain features are extracted through the spatial domain electronic branch 1051 and the frequency domain electronic branch 1052, respectively, and the cross-domain fusion unit 1053 performs feature fusion. This allows the local structural information in the spatial domain encoded image to complement and enhance the global periodicity, texture, and directionality information in the frequency domain encoded image at the network level. Compared to the spatial domain convolutional neural network, which requires stacking deep networks or expanding convolutional kernels to extract global frequency domain information, the scheme described in this disclosure enables the target neural network 105 to obtain rich perceptual information with a shallower network structure through explicit fusion of dual-domain features. This reduces the number of network parameters and computational load. At the same time, the complementarity of dual-domain information also helps to improve the accuracy and generalization ability of the task processing results.

[0064] Since spatial domain features and frequency domain features correspond to different physical coordinate systems, they cannot be directly spliced ​​together using only the same two-dimensional coordinates. Therefore, the scheme described in this disclosure introduces a cross-domain fusion unit 1053, which can fuse spatial domain features and frequency domain features, and generate task processing results based on the fusion result.

[0065] In some embodiments of this disclosure, the fusion result may include: a fusion result obtained by fusing spatial domain features and frequency domain features according to the fusion method corresponding to the task type corresponding to the input light field.

[0066] For example, the fusion methods may include: vector-level fusion, gated fusion, cross-domain attention fusion, and hierarchical fusion. If the task type corresponding to the input light field is image classification, scene recognition, quality detection, etc., vector-level fusion or gated fusion can be used; if the task type corresponding to the input light field is object detection, semantic segmentation, key point localization, hierarchical fusion can be used; if the task type corresponding to the input light field is fine-grained recognition or texture classification, cross-domain attention fusion can be used.

[0067] Accordingly, selecting the appropriate fusion method based on different task types can match the fusion strategy of spatial domain features and frequency domain features with the task requirements, avoiding the insufficient adaptability of a single fusion method in different task scenarios, and improving the processing accuracy and generalization ability of the target neural network 105 in different tasks.

[0068] In the vector-level fusion method, spatial domain features and frequency domain features can be converted into feature vectors through pooling or projection modules, and then the two feature vectors can be concatenated or weighted to obtain the fusion result.

[0069] In the gated fusion method, spatial domain features and frequency domain features can be converted into feature vectors through pooling or projection modules, and the importance of spatial domain features and frequency domain features can be automatically determined. The two feature vectors are dynamically weighted and combined through adaptive fusion weights to obtain the fusion result.

[0070] In cross-domain attention fusion, spatial domain features can be used as queries, and frequency domain features as keys and values. The attention mechanism calculates information related to spatial domain features in the frequency domain features and fuses this information with the spatial domain features. Alternatively, the frequency domain features can be set as queries, and spatial domain features as keys and values, forming bidirectional cross-domain attention to enhance the interaction and complementarity between the two domain features.

[0071] In the hierarchical fusion approach, spatial domain features and frequency domain features can be fused at multiple scale levels of the target neural network 105. Each layer extracts features from the spatial domain and frequency domain features output by the previous layer, and then the fusion module at the corresponding level completes the inter-layer feature fusion. For tasks requiring spatial localization, such as target detection, semantic segmentation, and keypoint localization, the spatial domain electronic branch 1051 can be used as the main branch, and the frequency domain electronic branch 1052 can be used as the global modulation branch. The frequency domain electronic branch 1052 outputs channel weights, scale weights, or global conditional vectors. These conditions are used to perform conditional modulation on the spatial domain features. This can be achieved using either scaling and offset modulation or feature multiplication and superposition modulation. While preserving the spatial domain feature positional structure, the frequency domain features are used to enhance the target neural network 105's perception of texture, scale, and periodic structure.

[0072] In practical applications, the optical encoder 101 can be jointly optimized with the target neural network 105.

[0073] Accordingly, Figure 5 This is a flowchart illustrating an embodiment of the training method for the optoelectronic hybrid computing system described in this disclosure. Figure 5 As shown, the specific implementation methods are as follows.

[0074] In step 501, training data is acquired.

[0075] In step 502, the optoelectronic hybrid computing system is trained based on the training data.

[0076] The optoelectronic hybrid computing system can provide Figure 1 or Figure 2 The system shown.

[0077] In some embodiments of this disclosure, training the optoelectronic hybrid computing system based on training data may include: initializing the parameters of an optical encoder (such as a metasurface optical encoder) in the optoelectronic hybrid computing system; fixing the optical encoder, frequency domain electronic branch, and cross-domain fusion unit, and training the spatial domain electronic branch using training data, wherein the spatial domain electronic branch, frequency domain electronic branch, and cross-domain fusion unit are all located in the target neural network of the optoelectronic hybrid computing system; in response to reaching the training termination condition, fixing the optical encoder and spatial domain electronic branch, and jointly training the frequency domain electronic branch and cross-domain fusion unit based on the training data, or fixing the optical encoder and partially fixing the spatial domain electronic branch, and jointly training the spatial domain electronic branch, frequency domain electronic branch, and cross-domain fusion unit based on the training data; in response to reaching the training termination condition again, jointly training the optical encoder and each component in the target neural network based on the training data.

[0078] For example, a differentiable optical model of the input light field, metasurface propagation, beam splitting, imaging, and Fourier transform can be established, and the parameters of the optical encoder can be initialized. Then, the optical encoder, frequency domain electronic branch, and cross-domain fusion unit can be fixed, and the spatial domain electronic branch can be trained using training data to obtain a stable baseline. The training termination condition can be reaching a predetermined number of training rounds. Afterwards, the optical encoder can be fixed, and the spatial domain electronic branch can be fixed or partially fixed. The frequency domain electronic branch and cross-domain fusion unit can be added for joint training, and the training termination condition can again be reaching a predetermined number of training rounds. Furthermore, based on the training data, joint training can be performed on the optical encoder and the various components (spatial domain electronic branch, frequency domain electronic branch, and cross-domain fusion unit) in the target neural network, i.e., end-to-end optoelectronic joint training. During end-to-end optoelectronic joint training, manufacturing errors, optical axis misalignment, focal length errors, sensor noise, and exposure variations can also be incorporated for robust training.

[0079] Each training stage can use corresponding training data. After training is complete, metasurface devices can be fabricated and actual optical paths can be built. If needed, the target neural network can be retrained or domain-adapted using real measurement data.

[0080] As can be seen, the scheme described in this disclosure can employ a phased training strategy to train the optoelectronic hybrid computing system. By training the spatial domain electronic branch separately first, the spatial domain electronic branch can converge to a better solution under a stable supervision signal, avoiding the training instability problem caused by the simultaneous random initialization of branches in the optical encoder and the target neural network. On this basis, the trained spatial domain electronic branch is fixed or partially fixed, and the frequency domain electronic branch and cross-domain fusion unit are trained together. This allows the frequency domain electronic branch to be trained with the stable features provided by the spatial domain electronic branch as a reference, effectively alleviating the mutual interference of dual-domain features in the initial stage and reducing the difficulty of joint training. Finally, end-to-end joint training is performed to achieve a globally optimal match between the structural parameters of the optical encoder and the network parameters of the target neural network, thereby fully leveraging the synergistic effect between the optical encoder and the target neural network and effectively improving the task processing performance and generalization ability of the entire system.

[0081] In some embodiments of this disclosure, when jointly training the components of the optical encoder and the target neural network, the corresponding task loss and the predetermined constraint loss can be obtained for the training data, and the comprehensive loss can be determined based on the task loss and the constraint loss. Then, the parameters of the components of the optical encoder and the target neural network can be updated based on the comprehensive loss.

[0082] The joint optimization objective can be: to minimize the task loss function of the entire system on the training dataset by simultaneously optimizing the structural parameters of the optical encoder and the network parameters of the target neural network. Specifically, the input light field is encoded by the optical encoder, resulting in a spatial domain encoded image and a frequency domain encoded image through spatial and frequency domain optical path branches, respectively. Both images are input into the target neural network. The combined loss can be calculated based on the network output and the ground truth label. The structural parameters of the optical encoder and the network parameters of the target neural network are then updated simultaneously using a backpropagation algorithm.

[0083] The overall loss can be: Ltotal=Ltask+λ1Lfab+λ2Lenergy+λ3Lrobust+λ4Ldiv; (8) Where Ltotal represents the overall loss, Lfab represents the metasurface manufacturability constraint loss, Lenergy represents the light energy utilization constraint loss, Lrobust represents the assembly error and noise robustness constraint loss, Ldiv represents the spatial and frequency domain feature complementarity constraint loss, and the complementarity constraint can be used to avoid the two branches learning exactly the same information. λ1, λ2, λ3 and λ4 represent the predetermined weight coefficients, respectively.

[0084] The method for obtaining the above-mentioned constraint losses can be determined according to actual needs. For example, Lfab can be calculated based on the difference between the structural parameters of the subwavelength structural units of the metasurface and the process design rules. If the structural parameters exceed the manufacturable range, this loss increases, and it is used to constrain the optical encoder to meet the processing requirements during design. Lenergy can be calculated based on the ratio of the optical power received by the first and second photodetectors to the input optical power. The lower the light energy utilization rate, the greater the loss, and it is used to constrain the system design to minimize light energy loss. Lrobust can be calculated by applying random perturbations to the structural parameters of the optical encoder or the optical path parameters of the system during training, based on the difference in system task performance before and after the perturbation. The greater the performance difference, the greater the loss, and it is used to enhance the system's tolerance to various non-ideal factors. Ldiv can be calculated based on the correlation between the spatial domain characteristics of the output of the spatial domain electronic branch and the frequency domain characteristics of the output of the frequency domain electronic branch. The higher the correlation, the greater the loss, and it is used to encourage the two branches to extract complementary rather than redundant features.

[0085] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. Furthermore, for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0086] The task processing results described in the embodiments of this disclosure are not targeted at any specific user and are not intended to reflect the personal information of any specific user. The collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, involved in the technical solutions of this disclosure comply with relevant laws and regulations and do not violate public order and good morals.

[0087] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0088] Figure 6A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0089] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output interface 605 is also connected to the bus 604.

[0090] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0091] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0092] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0093] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0094] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0095] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0096] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0097] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0098] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0099] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A hybrid optoelectronic computing system, characterized in that, include: Optical encoder, beam splitter module, spatial domain optical path branch, frequency domain optical path branch, and target neural network; The optical encoder is used to optically encode the input light field to obtain the encoded light field; The beam splitting module is used to split the coded light field into a first beam and a second beam; The spatial domain optical path branch is used to generate a spatial domain encoded image based on the first beam. The frequency domain optical path branch is used to generate a frequency domain encoded image based on the second beam. The target neural network is used to generate the task processing result corresponding to the input light field based on the spatial domain encoded image and the frequency domain encoded image.

2. The system according to claim 1, characterized in that, The optical encoder includes: a metasurface optical encoder; The metasurface optical encoder employs a static metasurface and / or a tunable metasurface.

3. The system according to claim 1, characterized in that, The spatial domain optical path branch includes: a spatial domain optical module and a first photodetector; the spatial domain optical module is used to perform spatial domain optical processing on the first beam to obtain a first processing result, and the first photodetector is used to generate the spatial domain encoded image based on the first processing result; The frequency domain optical path branch includes: a frequency domain optical module and a second photodetector; the frequency domain optical module is used to perform frequency domain optical processing on the second beam to obtain a second processing result, and the second photodetector is used to generate the frequency domain encoded image based on the second processing result.

4. The system according to claim 3, characterized in that, The frequency domain optical path branch also includes: a frequency domain modulator; The frequency domain modulator is used to perform frequency domain modulation on the second processing result to obtain a modulation result; The second photodetector is further used to generate the frequency domain coded image based on the modulation result.

5. The system according to claim 3, characterized in that, The second photodetector is further configured to generate an interference fringe image based on the interference result of the reference beam and the second processing result, wherein the reference beam is an inclined beam positioned in front of the frequency domain optical module; The frequency domain optical path branch further includes a post-processing unit, used to generate the frequency domain coded image based on the interference fringe image.

6. The system according to claim 1, characterized in that, The target neural network includes: a spatial domain electronic branch, a frequency domain electronic branch, and a cross-domain fusion unit; The spatial domain electronic branch is used to extract spatial domain features from the spatial domain encoded image; The frequency domain electronic branch is used to extract frequency domain features from the frequency domain encoded image; The cross-domain fusion unit is used to fuse the spatial domain features and the frequency domain features, and generate the task processing result based on the fusion result.

7. The system according to claim 6, characterized in that, The fusion result includes: the fusion result obtained by fusing the spatial domain features and the frequency domain features according to the fusion method corresponding to the task type corresponding to the input light field.

8. A training method for a hybrid optoelectronic computing system, characterized in that, include: Obtain training data; The optoelectronic hybrid computing system is trained based on the training data, and the optoelectronic hybrid computing system is the system described in any one of claims 1 to 7.

9. The method according to claim 8, characterized in that, The step of training the optoelectronic hybrid computing system based on the training data includes: Initialize the parameters of the optical encoder in the optoelectronic hybrid computing system; The optical encoder, frequency domain electronic branch, and cross-domain fusion unit are fixed, and the spatial domain electronic branch is trained using the training data. The spatial domain electronic branch, the frequency domain electronic branch, and the cross-domain fusion unit are all located in the target neural network of the optoelectronic hybrid computing system. In response to reaching the training termination condition, the optical encoder and the spatial domain electronic branch are fixed, and the frequency domain electronic branch and the cross-domain fusion unit are jointly trained according to the training data; or, the optical encoder is fixed, and the spatial domain electronic branch is partially fixed, and the spatial domain electronic branch, the frequency domain electronic branch and the cross-domain fusion unit are jointly trained according to the training data. In response to the training termination condition being met again, the optical encoder and the components of the target neural network are jointly trained based on the training data.

10. The method according to claim 9, characterized in that, The joint training of the components of the optical encoder and the target neural network includes: For the training data, the corresponding task loss and the predetermined constraint loss are obtained respectively. A comprehensive loss is determined based on the task loss and the constraint loss. The parameters of each component in the optical encoder and the target neural network are updated based on the comprehensive loss.

11. A training device for a hybrid optoelectronic computing system, characterized in that, include: Data acquisition module and system training module; The data acquisition module is used to acquire training data; The system training module is used to train the optoelectronic hybrid computing system based on the training data, wherein the optoelectronic hybrid computing system is the system described in any one of claims 1 to 7.

12. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 8-10.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 8-10.

Images