Method and apparatus for training and task processing result generation of optoelectronic hybrid computing system

Abstract

The present disclosure provides a training method and a task processing result generation method of an optoelectronic hybrid computing system, and relates to the fields of artificial intelligence and optoelectronic imaging. The training method comprises: obtaining a training sample, wherein the training sample comprises a training image and a true label, and the true label is a task processing result corresponding to the training image; for the training image, generating C channel scalars in parallel by using an optical front end, wherein C is a positive integer greater than 1, the optical front end corresponds to the C channels, each channel scalar represents a physical response of the corresponding channel to the training image, and the optoelectronic hybrid computing system comprises the optical front end and an electronic back end; generating the task processing result corresponding to the training image by using the electronic back end according to the C channel scalars, and determining the task processing result as a predicted label; determining a comprehensive loss according to the predicted label and the true label, and updating parameters of the optical front end and / or the electronic back end according to the comprehensive loss. By using the scheme of the present disclosure, the power consumption and delay can be reduced.

Description

Technical Field

[0001] This disclosure relates to fields such as artificial intelligence and optical imaging, particularly to technologies such as computational imaging, deep learning, and intelligent sensing, and especially to methods and apparatus for generating training and task processing results of optoelectronic hybrid computing systems. Background Technology

[0002] In traditional image classification and object detection scenarios, an architecture of optical imaging + electronic computing is usually adopted. The optical front end is responsible for forming the image, and its function is close to a single convolutional layer. The subsequent heavy visual tasks (such as classification) are all handled by the deep convolutional neural network (CNN) in the electronic back end, which increases power consumption and latency. Summary of the Invention

[0003] This disclosure provides a method and apparatus for generating training and task processing results in a hybrid optoelectronic computing system.

[0004] A training method for a hybrid optoelectronic computing system includes: Obtain training samples, which include: training images and real labels, wherein the real labels are the task processing results corresponding to the training images; For the training image, C channel scalars are generated in parallel using an optical front end, where C is a positive integer greater than 1. The optical front end corresponds to C channels, and each channel scalar represents the physical response of the corresponding channel to the training image. The optoelectronic hybrid computing system includes the optical front end and the electronic back end. Based on the C channel scalars, the electronic backend is used to generate the task processing result corresponding to the training image and determine it as the predicted label; The comprehensive loss is determined based on the predicted label and the actual label, and the parameters of the optical front end and / or the electronic back end are updated based on the comprehensive loss.

[0005] A method for generating task processing results based on a hybrid optoelectronic computing system includes: For the target scene, C channel scalars are generated in parallel using an optical front end, where C is a positive integer greater than 1. The optical front end corresponds to C channels, and each channel scalar represents the physical response of the corresponding channel to the target scene. The optoelectronic hybrid computing system includes the optical front end and the electronic back end. Based on the C channel scalars, the electronic backend generates the task processing result corresponding to the target scenario.

[0006] A training device for a hybrid optoelectronic computing system includes: a sample acquisition module, a front-end processing module, a back-end processing module, and a parameter update module; The sample acquisition module is used to acquire training samples, which include training images and real labels, wherein the real labels are the task processing results corresponding to the training images. The front-end processing module is used to generate C channel scalars in parallel using an optical front-end for the training image, where C is a positive integer greater than 1. The optical front-end corresponds to C channels, and each channel scalar represents the physical response of the corresponding channel to the training image. The optoelectronic hybrid computing system includes the optical front-end and the electronic back-end. The back-end processing module is used to generate the task processing result corresponding to the training image based on the C channel scalars using the electronic back-end, and determine it as the prediction label; The parameter update module is used to determine the comprehensive loss based on the predicted label and the real label, and update the parameters of the optical front end and / or the electronic back end based on the comprehensive loss.

[0007] A hybrid optoelectronic computing system includes: an optical front-end and an electronic back-end; The optical front end is used to generate C channel scalars in parallel for the target scene, where C is a positive integer greater than 1. The optical front end corresponds to C channels, and each channel scalar represents the physical response of the corresponding channel to the target scene. The electronic backend is used to generate the task processing result corresponding to the target scenario based on the C channel scalars.

[0008] An electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0009] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0010] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating an embodiment of the training method for the optoelectronic hybrid computing system disclosed herein; Figure 2 This is a flowchart of an embodiment of the task processing result generation method based on an optoelectronic hybrid computing system disclosed herein; Figure 3 This is a schematic diagram of the composition structure of a training device embodiment of the optoelectronic hybrid computing system disclosed herein. Figure 4 This is a schematic diagram of the composition structure of the first embodiment of the optoelectronic hybrid computing system disclosed herein; Figure 5 This is a schematic diagram of the composition structure of the second embodiment of the optoelectronic hybrid computing system disclosed herein; Figure 6 A schematic block diagram of an electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0015] Figure 1 This is a flowchart illustrating an embodiment of the training method for the optoelectronic hybrid computing system described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0016] In step 101, training samples are obtained, which include training images and real labels, where the real labels are the task processing results corresponding to the training images.

[0017] In step 102, for the training image, C channel scalars are generated in parallel using an optical front end, where C is a positive integer greater than 1. The optical front end corresponds to C channels, and each channel scalar represents the physical response of the corresponding channel to the training image. The optoelectronic hybrid computing system includes an optical front end and an electronic back end.

[0018] In step 103, based on the C channel scalars, the task processing results corresponding to the training images are generated using the electronic backend and determined as the predicted labels.

[0019] In step 104, the overall loss is determined based on the predicted label and the actual label, and the parameters of the optical front end and / or electronic back end are updated based on the overall loss.

[0020] By employing the scheme described in the above embodiments, end-to-end joint training (joint optimization) of the optical front-end and electronic back-end can be performed, enabling them to collaboratively adapt to specific tasks and thereby improving task processing performance at the system level. Furthermore, in practical applications, computationally intensive pixel-level multiply-accumulate operations in image processing can be transferred to the optical front-end for physical parallel processing, directly outputting low-dimensional channel scalars. In this way, the electronic back-end only needs to perform some simple processing on C channel scalars to obtain the required task processing results, thus reducing the computational burden on the electronic back-end and lowering power consumption and latency.

[0021] The Kolmogorov Superposition Theorem (KST) states that any multivariate continuous function can be obtained by superimposing a finite number of univariate functions, but it lacks an engineered, trainable physical implementation. The scheme described in this disclosure borrows the idea of ​​KST, transforming the traditional optical imaging + electronic computation into optical feature extraction + electronic decision-making, shifting the computational burden from the electronic domain to the optical physical domain, and utilizing the parallelism of the optical physical domain to achieve low power consumption. Structurally, it implements a representation consistent with the KST form: "univariate function (which can be replaced with a few-variate local function in this scheme) + summation + composition," which can be used in different scenarios such as image classification and object detection.

[0022] The scheme described in this disclosure does not impose any restrictions on how to construct training samples. For example, taking image classification as an example, training images can come from a subset of natural images and need to include as much optical and physical information as possible. The true label refers to the image classification result, which can be manually generated or automatically generated through some method. Using the training samples, the optical front-end and electronic back-end of the optoelectronic hybrid computing system described in this disclosure can be jointly trained end-to-end.

[0023] In the scheme described in this disclosure, the optical front end can be an optical front end based on a metasurface array. The metasurface array can be divided into Q patches. The C channels include: C channels determined according to the Q patches and a predetermined optical front end configuration. The optical front end configuration may include: S polarization configurations and L hyperspectral discrete band configurations, or the optical front end configuration may include: S polarization configurations, L hyperspectral discrete band configurations and T time-division multiplexing configurations. Q, S, L and T are all positive integers greater than 1, and their specific values ​​can be determined according to actual needs.

[0024] The channel described in this disclosure is an optical information acquisition path uniquely determined by multi-dimensional configurations such as space, polarization, spectrum, and time. The channel scalar is the numerical result corresponding to the final output of each channel.

[0025] A metasurface is an artificial layered material with a size smaller than or approximately equal to the wavelength. It can be considered a two-dimensional counterpart of a metamaterial. By using subwavelength metastructural units on its surface, it is possible to control the polarization, phase, amplitude, frequency, and propagation mode of electromagnetic waves, achieving properties such as beam shaping, beam deflection, superlensing, superholography, optical rotation, and anti-reflection / anti-reflection enhancement. The metasurface array in the scheme described in this disclosure can be divided into Q = R * R blocks, where R is a positive integer greater than 1. Each block can have programmable phase and amplitude responses, forming a spatially variable point spread function (PSF). The PSF determines the specific diffused light spot formed on the image sensor by a point light source in the scene after passing through this block.

[0026] Assuming R=10, then Q=100. Furthermore, assuming S=4 (0°, 45°, 90°, 135° linear polarization analysis), L=8 (8 discrete bands), and T=2 (2 sets of phase masks), then without considering time-division multiplexing, the number of channels C can be calculated by combining Q, S, and L: C=Q*S*L=3200. If time-division multiplexing is considered, the number of channels C can be calculated by combining Q, S, L, and T: C=Q*S*L*T=6400. Preferably, S polarization configurations, L hyperspectral discrete band configurations, and T time-division multiplexing configurations can be performed simultaneously, resulting in 6400 channels. By introducing time-division multiplexing, the number of channels can be multiplied by switching the metasurface configuration in the time dimension without increasing physical hardware. This is equivalent to improving the feature representation capability of the optoelectronic hybrid computing system at the cost of exposure time.

[0027] In some embodiments of this disclosure, updating the parameters of the optical front end may include updating the structural parameters of the metasurface array and the function parameters of the readout window functions corresponding to the C channels, wherein the readout window functions are used to convert the optical modulation response of the corresponding channel into a channel scalar.

[0028] In addition, in some embodiments of this disclosure, the method of generating task processing results corresponding to training images using the electronic backend and determining them as predicted labels may include: performing nonlinear transformations on each channel scalar using the corresponding outer unary function to obtain transformation results; combining the transformation results to generate original processing results using a linear decision function; and normalizing the original processing results using a normalized exponential function (softmax) to obtain predicted labels. Accordingly, the method of updating the parameters of the electronic backend may include: updating the function parameters of the outer unary function and the linear decision function.

[0029] In other words, the scheme described in this disclosure proposes a KST-style optoelectronic hybrid computing system based on metasurfaces, which can realize the coupling of programmable metasurfaces and electronically learnable modules. Taking advantage of the fact that real images / targets are usually located on low-dimensional manifolds or low-rank / sparse structures, the main computation of high-dimensional vision tasks is moved forward to the optical front end. It achieves trainable, interpretable and energy-efficient inference and learning in the form of "channel unary + electronic unary outer layer", which is suitable for edge devices, low-power sensing and fast intelligent perception and other scenarios.

[0030] Corresponding to the inner and outer layer structures of KST, the inner layer of the scheme described in this disclosure is implemented by the PSF of the metasurface block and the readout window function (a readout function that selects certain components in the sensor plane and performs weighted summation), and the outer layer is implemented by the outer layer unary function, and can combine the transformation results corresponding to each channel to generate task processing results, etc.

[0031] The optical front end is equivalent to performing the following operations in parallel: (1) This represents the channel scalar, where P represents the total number of pixels (or positions) in the local light field involved in calculating this channel scalar, determined by the support region of the PSF. Represents the light field point, This refers to a predefined inner unary function, indicating that... The applied physical transformation The equivalent weight of the optical link is represented by the contribution weight of the light field point at position p to the channel c, where 1 ≤ c ≤ C.

[0032] The above formula mathematically describes the behavior of the readout window function, whose parameters can be updated during training. Additionally, the structural parameters of the metasurface array can also be updated; specifically, this refers to updating the nanostructure parameters of each block, which may include the length, width, and rotation angle of the nanostructure.

[0033] The electronic back-end receives C channels of scalar data from the optical front-end. Then, the corresponding outer univariate functions can be used in parallel to perform nonlinear transformations on the scalars of each channel, thereby obtaining C transformation results. Then, the C transformation results can be combined to generate the original processing results using a linear decision function, and the original processing results can be normalized using softmax to obtain the predicted task processing results (predicted labels).

[0034] That is: (2) in, The outermost univariate function refers to a learnable nonlinear transformation applied to a single-channel scalar at the electron back end. The specific form can be determined according to actual needs, such as the requirements of the task. This indicates the result of the transformation.

[0035] Then, all transformation results can be summarized, and the original processing result can be generated using a linear decision function. This allows for final linear readout, summarizing the transformation results from the C outer unary functions, and mapping them back to the original processing result using a trainable linear transformation. Furthermore, softmax can be used to normalize the original processing result, thereby obtaining the desired predicted label.

[0036] Through the above processing, a large number of multiply-accumulate operations can be transferred to the optical front end to be completed in parallel in a physical manner. The electronic back end only needs to perform a small number of lightweight calculations for channel scalars, which greatly reduces the amount of computation in the electronic back end, and thus reduces the demand for computing power and memory.

[0037] In some embodiments of this disclosure, the outer univariate function can be implemented using B-splines or a small multilayer perceptron (MLP). A B-spline is a parameterized curve fitting function that uses a set of adjustable parameters called control points to smoothly approximate a complex objective function through a linear combination of basis functions. It is computationally efficient and flexible. A small MLP refers to a very small neural network, such as a neural network with only one hidden layer and a few neurons, but which needs to have strong nonlinear fitting capabilities.

[0038] After obtaining the predicted labels, the comprehensive loss can be determined based on the predicted labels and the true labels. In some embodiments of this disclosure, the task loss can be determined based on the predicted labels and the true labels, and M auxiliary losses can be obtained, where M is a positive integer. Then, the comprehensive loss can be determined based on the task loss, each auxiliary loss, and the weight coefficients corresponding to each auxiliary loss.

[0039] Specifically, in some embodiments of this disclosure, a smoothing regularization term can be generated based on the smoothness of each outer unary function, and a physical regularization term can be generated based on the difference between the structural parameters of the metasurface array and the physical manufacturing constraints. Furthermore, an orthogonal regularization term can be determined based on the similarity between each outer unary function. In this way, the obtained smoothing regularization term, physical regularization term, and orthogonal regularization term can be determined as the required auxiliary loss.

[0040] That is: (3) Where L represents the overall loss, The task loss can be represented by, for example, the difference between the predicted label and the true label can be calculated using cross-entropy loss, and this difference can be determined as the task loss. This represents a smoothing regularization term, used to prevent overfitting of the outer univariate function. This represents the square of the gradient, 1≤c≤C. This represents a physical regularization term used to ensure that the optimized metasurface's structural parameters are physically manufacturable and functional; the physical manufacturing constraints can be predetermined. The orthogonality regularization term is obtained by calculating the similarity between different outer univariate functions. Its purpose is to penalize the similarity between the outer univariate functions corresponding to different channels, reducing redundancy between channels and improving the expressive power of the channels. These represent weighting coefficients, both of which are hyperparameters. Their specific values ​​can be preset to control the importance of the corresponding regularization terms in the overall loss calculation.

[0041] In practical applications, the comprehensive loss can also be generated based on the task loss and one or any combination of the auxiliary losses. However, it is preferable to calculate the comprehensive loss using the method in formula (3).

[0042] In summary, by adopting the above processing method, auxiliary loss can be introduced to form a comprehensive loss together with task loss, which can effectively constrain the optimization direction. This not only improves the accuracy of task processing, but also ensures that the optimized parameters meet constraints such as physical realizability, function smoothness, and channel independence, thus significantly improving the optimization effect.

[0043] In practical applications, the training process can be terminated once the convergence condition is met.

[0044] In addition, the training process can be implemented in the following two ways: 1. Fully digital simulation method: Based on electromagnetic wave propagation theory and optical system modeling, an end-to-end differentiable simulator for metasurface modulation, light field propagation and sensor response is built in a computing environment. Under this framework, all optical parameters (such as the structural parameters of the metasurface) and electronic parameters (such as the function parameters of the outer univariate function) can be jointly optimized by the gradient descent algorithm. The training process is completed entirely in a virtual environment, and the final optimization results can be used to guide physical deployment.

[0045] 2. Physical hardware approach: The programmable metasurface is directly used as the optical front end and embedded into the training loop. The system response (such as channel scalar) of the metasurface under different configurations is measured in real time. Combined with a differentiable electronic back end, end-to-end gradient estimation and parameter updates are performed until the final optimization result is obtained.

[0046] Based on the final optimization results, the actual fabrication of the optical front end and electronic back end can be carried out to obtain a photoelectric hybrid computing system. The photoelectric hybrid computing system can then be applied to perform actual task processing to generate the required task processing results for the target scenario.

[0047] Accordingly, Figure 2 This is a flowchart illustrating an embodiment of the task processing result generation method based on a photoelectric hybrid computing system described in this disclosure. Figure 2 As shown, the specific implementation methods are as follows.

[0048] In step 201, for the target scene, C channel scalars are generated in parallel using the optical front end, where C is a positive integer greater than 1. The optical front end corresponds to C channels, and each channel scalar represents the physical response of the corresponding channel to the target scene. The optoelectronic hybrid computing system includes an optical front end and an electronic back end.

[0049] In step 202, based on the C channel scalars, the electronic backend is used to generate the task processing results corresponding to the target scenario.

[0050] The parameters of the optical front end and the electronic back end can be determined according to... Figure 1 The method shown is used for training.

[0051] In addition, the method of generating task processing results corresponding to the target scene using the electronic backend may include: performing nonlinear transformation on each channel scalar using the corresponding outer univariate function to obtain the transformation result, combining the transformation results, generating the original processing result using a linear decision function, and normalizing the original processing result using softmax to obtain the task processing result.

[0052] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. Furthermore, for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0053] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0054] Figure 3 This is a schematic diagram of the structural composition of an embodiment 300 of the optoelectronic hybrid computing system described in this disclosure. Figure 3 As shown, it includes: a sample acquisition module 301, a front-end processing module 302, a back-end processing module 303, and a parameter update module 304.

[0055] The sample acquisition module 301 is used to acquire training samples, which include training images and real labels, wherein the real labels are the task processing results corresponding to the training images.

[0056] The front-end processing module 302 is used to generate C channel scalars in parallel for the training image using the optical front end, where C is a positive integer greater than 1. The optical front end corresponds to C channels, and each channel scalar represents the physical response of the corresponding channel to the training image. The optoelectronic hybrid computing system includes an optical front end and an electronic back end.

[0057] The back-end processing module 303 is used to generate the task processing results corresponding to the training images based on the C channel scalars using the electronic back-end, and determine them as prediction labels.

[0058] The parameter update module 304 is used to determine the comprehensive loss based on the predicted label and the actual label, and to update the parameters of the optical front end and / or electronic back end based on the comprehensive loss.

[0059] In some embodiments of this disclosure, the optical front end may be an optical front end based on a metasurface array. The metasurface array may be divided into Q blocks, and the C channels may include: C channels determined according to the Q blocks and a predetermined optical front end configuration. The optical front end configuration may include: S polarization configurations and L hyperspectral discrete band configurations, or the optical front end configuration may include: S polarization configurations, L hyperspectral discrete band configurations and T time-division multiplexing configurations, where Q, S, L and T are all positive integers greater than 1.

[0060] In some embodiments of this disclosure, the parameter update module 304 may update the parameters of the optical front end by updating the structural parameters of the metasurface array and the function parameters of the readout window functions corresponding to the C channels, wherein the readout window functions are used to convert the optical modulation response of the corresponding channel into a channel scalar.

[0061] In some embodiments of this disclosure, the back-end processing module 303 generates task processing results corresponding to training images using the electronic back-end and determines them as predicted labels in the following ways: for each channel scalar, a nonlinear transformation is performed using the corresponding outer univariate function to obtain the transformation result; the original processing result is generated using a linear decision function by combining the transformation results; and the original processing result is normalized using softmax to obtain the predicted label. Correspondingly, the parameter update module 304 updates the parameters of the electronic back-end in the following ways: updating the function parameters of the outer univariate function and the linear decision function.

[0062] In some embodiments of this disclosure, the outer unary function may be implemented using B-splines or a small MLP.

[0063] In addition, in some embodiments of this disclosure, the parameter update module 304 may determine the comprehensive loss based on the predicted label and the true label in the following ways: determine the task loss based on the predicted label and the true label, and obtain M auxiliary losses, where M is a positive integer, and determine the comprehensive loss based on the task loss, each auxiliary loss and the weight coefficient corresponding to each auxiliary loss.

[0064] Specifically, in some embodiments of this disclosure, the parameter update module 304 may obtain M auxiliary losses in the following ways: generating smoothing regularization terms based on the smoothness of each outer unary function, generating physical regularization terms based on the difference between the structural parameters of the metasurface array and the physical manufacturing constraints, determining orthogonal regularization terms based on the similarity between each outer unary function, and determining the smoothing regularization terms, physical regularization terms, and orthogonal regularization terms as auxiliary losses.

[0065] Figure 4 This is a schematic diagram of the structural composition of the first embodiment 400 of the optoelectronic hybrid computing system described in this disclosure. Figure 4 As shown, it includes: an optical front end 41 and an electronic back end 42.

[0066] Optical front-end 41 is used to generate C channel scalars in parallel for the target scene, where C is a positive integer greater than 1. The optical front-end corresponds to C channels, and each channel scalar represents the physical response of the corresponding channel to the target scene.

[0067] The electronic backend 42 is used to generate the task processing results corresponding to the target scenario based on the C channel scalars.

[0068] The parameters of the optical front end 41 and the electronic back end 42 can be determined according to... Figure 1 The method shown is used for training.

[0069] Figure 5 This is a schematic diagram of the structural composition of the second embodiment 500 of the optoelectronic hybrid computing system described in this disclosure. Figure 5 As shown, it includes an optical front end 41 and an electronic back end 42. The optical front end 41 may include a metasurface array 411, a polarization analyzer 412, a hyperspectral selection module 413, an image sensor 414, and a sensor readout module 415. The electronic back end 42 may include a digital processing module 421. In addition, the optoelectronic hybrid computing system 500 may also include a metasurface driving and control module 43 and a clock and synchronization control module 44.

[0070] The metasurface array 411 can be divided into Q blocks, the polarization analyzer 412 can be used to implement S different polarization configurations, the hyperspectral selection module 413 can be used to implement L different hyperspectral discrete band configurations, the metasurface array 411, the polarization analyzer 412 and the hyperspectral selection module 413 can work together to modulate the input light field, the image sensor 414 can be used to receive the modulated light field, and can form pixel values ​​corresponding to the light intensity distribution through photoelectric conversion and integration, and the sensor readout module 415 can be used to generate the channel scalar of the corresponding channel by weighted summation of the pixel values ​​of the corresponding region using the readout window function corresponding to each channel, where Q, S and T are all positive integers greater than 1.

[0071] The metasurface array 411 can be a planar optical element based on silicon or silicon nitride, on which millions of subwavelength nanostructures (such as nanofins and nanopillars) are fabricated. The metasurface array 411 can be divided into R×R blocks, each block can include multiple nanostructures, and each block can realize a corresponding PSF. In addition, the metasurface array 411 can be one or more stacked metasurface arrays.

[0072] The polarization analyzer 412 can be located after the metasurface array 411 and before the image sensor 414. It can be a rotatable linear polarizer or a more advanced polarization camera chip, with micro / nano polarization filters integrated on each pixel. The polarization analyzer 412 can work in conjunction with the metasurface array 411. For each block in the metasurface array 411, the polarization analyzer 412 can allow selection of specific polarization states (such as 0°, 45°, 90°, 135°) for transmission, thereby extracting the polarization information of the scene.

[0073] The hyperspectral selection module 413 can be a tunable acousto-optic filter or a liquid crystal tunable filter, and can be placed before or after the metasurface array 411. Figure 5Taking the metasurface array 411 as an example, the hyperspectral selection module 413 is used for rapid switching of transmission bands. Alternatively, the hyperspectral selection module 413 can also be a multilayer diffractive optical element and can be integrated with the metasurface array 411, with its dispersion characteristics designed to produce different responses to different wavelengths. In addition, the hyperspectral selection module 413 can work in synergy with the metasurface array 411 and the polarization analyzer 412 in the spectral dimension to capture the optical field information of L discrete bands sequentially or in parallel.

[0074] Image sensor 414 can be used as a parallel integrator array, and each block in metasurface array 411 can correspond to a specific region on image sensor 414. Depending on the specific application scenario and performance requirements, image sensor 414 can be implemented using complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD). To further improve the system's performance in terms of information dimension, sensitivity, or speed, special sensors such as polarization sensors, snapshot spectral sensors, single-photon avalanche diode (SPAD) arrays, or pixel-locked arrays can also be used.

[0075] The sensor readout module 415 can be a readout integrated circuit (ROIC) built into the CMOS or an external analog-to-digital converter (ADC). It can use the readout window function corresponding to each channel to generate the channel scalar of the corresponding channel by weighted summation of the pixel values ​​of the corresponding area on the image sensor 414, rather than generating a complete image.

[0076] The digital processing module 421 in the electronic back-end 42 can be used to perform nonlinear transformations on each channel scalar using the corresponding outer univariate function to obtain the transformation results. It can also combine the transformation results to generate the original processing results using a linear decision function. Furthermore, it can use softmax to normalize the original processing results to obtain the task processing results corresponding to the target scene.

[0077] In other words, the digital processing module 421 can mathematically decode the signal optically encoded by the optical front end 41 to complete the task processing for the target scene. Depending on the system's requirements for power consumption and speed, the digital processing module 421 can be implemented using a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), or a Graphics Processing Unit (GPU). FPGAs can achieve a perfect combination of hardware parallel processing and field reconfigurability, and support rapid algorithm iteration and deployment. ASICs have advantages such as low power consumption and high speed, making them very suitable for mass production. GPUs can be low-power GPUs, which can significantly reduce power consumption while maintaining powerful parallel computing capabilities.

[0078] The metasurface driving and control module 43 can be used to drive the metasurface array 411 with voltage and can perform time-division multiplexing control on the metasurface array 411 according to T time-division multiplexing configurations, where T is a positive integer greater than 1. The clock and synchronization control module 44 can be used to perform timing synchronization control on each component in the optical front end 41.

[0079] The metasurface driving and control module 43 can be implemented using a high-channel-count digital-to-analog converter (DAC) board or an integrated driver chip. If the metasurface array 411 is a liquid crystal or phase change material, the metasurface driving and control module 43 can provide precise voltage to control the phase of each block. It can also quickly switch the configuration state of the entire metasurface array 411 within microseconds to milliseconds according to the preset T-set phase mask scheme.

[0080] The clock and synchronization control module 44, which can be implemented using an FPGA, is the system's "metronome" and is used to precisely synchronize the rotation or switching of the polarization analyzer 412, the tuning of the hyperspectral selection module 413, and the exposure and readout timing of the image sensor 414.

[0081] The metasurface driving and control module 43 and the clock and synchronization control module 44 can be set independently or integrated into the optical front end 41, which is very flexible and convenient.

[0082] In addition Figure 5 In addition to the example shown, in practical applications, the optical front end 41 may further include some other components, such as an optical lens group located in front of the metasurface array 411, for collecting and guiding light from the target scene and converging it onto the metasurface array 411.

[0083] The specific workflow of the above-described apparatus and system embodiments can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated hereafter.

[0084] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0085] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0086] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0087] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0088] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0089] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0090] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0091] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0092] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0093] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0094] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0095] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0096] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for a hybrid optoelectronic computing system, characterized in that, include: Obtain training samples, which include: training images and real labels, wherein the real labels are the task processing results corresponding to the training images; For the training image, C channel scalars are generated in parallel using an optical front end, where C is a positive integer greater than 1. The optical front end corresponds to C channels, and each channel scalar represents the physical response of the corresponding channel to the training image. The optoelectronic hybrid computing system includes the optical front end and an electronic back end. The optical front end is an optical front end based on a metasurface array, which is divided into Q blocks. The C channels include C channels determined according to the Q blocks and a predetermined optical front end configuration, where Q is a positive integer greater than 1. The optical front end includes the metasurface array and the following components: an image sensor for receiving a light field modulated from an input light field and forming pixel values ​​corresponding to the light intensity distribution through photoelectric conversion and integration; and a sensor readout module for performing the following processing for each channel: using the readout window function corresponding to the channel, weighted summing of the pixel values ​​of the corresponding region of the block corresponding to the channel on the image sensor, and determining the weighted sum as the channel scalar, wherein each block corresponds to a specific region on the image sensor. Based on the C channel scalars, the electronic backend is used to generate the task processing result corresponding to the training image and determine it as the prediction label. This includes: performing a nonlinear transformation on each channel scalar using the corresponding outer univariate function to obtain the transformation result; combining the transformation results and using a linear decision function to generate the original processing result; and using a normalized exponential function to normalize the original processing result to obtain the prediction label. The comprehensive loss is determined based on the predicted label and the actual label, and the parameters of the optical front end and / or the electronic back end are updated based on the comprehensive loss.

2. The method according to claim 1, characterized in that, The optical front-end configuration includes: S polarization configurations and L hyperspectral discrete band configurations, or the optical front-end configuration includes: S polarization configurations, L hyperspectral discrete band configurations and T time-division multiplexing configurations, where S, L and T are all positive integers greater than 1.

3. The method according to claim 1, characterized in that, Updating the parameters of the optical front end includes updating the structural parameters of the metasurface array and the function parameters of the readout window functions corresponding to the C channels, wherein the readout window functions are used to convert the optical modulation response of the corresponding channel into the channel scalar.

4. The method according to claim 3, characterized in that, Updating the parameters of the electronic backend includes updating the function parameters of the outer univariate function and the linear decision function.

5. The method according to claim 4, characterized in that, The outer univariate function is implemented using B-splines or a small multilayer perceptron.

6. The method according to claim 4, characterized in that, The determination of the comprehensive loss based on the predicted label and the true label includes: The task loss is determined based on the predicted label and the true label, and M auxiliary losses are obtained, where M is a positive integer; The comprehensive loss is determined based on the task loss, each auxiliary loss, and the weighting coefficients corresponding to each auxiliary loss.

7. The method according to claim 6, characterized in that, The acquisition of M auxiliary losses includes: Generate smoothing regularization terms based on the smoothness of each outer univariate function; A physical regularization term is generated based on the difference between the structural parameters of the metasurface array and the physical manufacturing constraints; Orthogonal regularization terms are determined based on the similarity between the outermost univariate functions; The smoothing regularization term, the physical regularization term, and the orthogonal regularization term are determined as the auxiliary loss.

8. A method for generating task processing results based on a photoelectric hybrid computing system, characterized in that, include: For the target scene, C channel scalars are generated in parallel using an optical front end, where C is a positive integer greater than 1. The optical front end corresponds to C channels, and each channel scalar represents the physical response of the corresponding channel to the target scene. The optoelectronic hybrid computing system includes the optical front end and the electronic back end. The optical front end is an optical front end based on a metasurface array, which is divided into Q blocks. The C channels include C channels determined according to the Q blocks and a predetermined optical front end configuration, where Q is a positive integer greater than 1. The optical front end includes the metasurface array and the following components: an image sensor for receiving a light field modulated by an input light field and forming pixel values ​​corresponding to the light intensity distribution through photoelectric conversion and integration; and a sensor readout module for performing the following processing for each channel: using the readout window function corresponding to the channel, weighted summing of the pixel values ​​of the corresponding area of ​​the block corresponding to the channel on the image sensor, and determining the weighted sum as the channel scalar, wherein each block corresponds to a specific area on the image sensor. Based on the C channel scalars, the electronic backend generates the task processing result corresponding to the target scene, including: performing a nonlinear transformation on each channel scalar using the corresponding outer univariate function to obtain the transformation result; combining the transformation results to generate the original processing result using a linear decision function; and normalizing the original processing result using a normalized exponential function to obtain the task processing result.

9. A training device for a hybrid optoelectronic computing system, characterized in that, include: The module includes a sample acquisition module, a front-end processing module, a back-end processing module, and a parameter update module. The sample acquisition module is used to acquire training samples, which include training images and real labels, wherein the real labels are the task processing results corresponding to the training images. The front-end processing module is used to generate C channel scalars in parallel using an optical front-end for the training image, where C is a positive integer greater than 1. The optical front-end corresponds to C channels, and each channel scalar represents the physical response of the corresponding channel to the training image. The optoelectronic hybrid computing system includes the optical front-end and the electronic back-end. The optical front-end is an optical front-end based on a metasurface array, which is divided into Q blocks. The C channels include C channels determined according to the Q blocks and a predetermined optical front-end configuration, where Q is a positive integer greater than 1. The optical front-end includes the metasurface array and the following components: an image sensor, used to receive a light field modulated by an input light field and form pixel values ​​corresponding to the light intensity distribution through photoelectric conversion and integration; and a sensor readout module, used to perform the following processing for each channel: using the readout window function corresponding to the channel, weighted summing of the pixel values ​​of the corresponding region of the block corresponding to the channel on the image sensor, and determining the weighted sum as the channel scalar, wherein each block corresponds to a specific region on the image sensor. The back-end processing module is used to generate the task processing result corresponding to the training image based on the C channel scalars using the electronic back-end, and determine it as the prediction label. The module includes: performing nonlinear transformation on each channel scalar using the corresponding outer univariate function to obtain the transformation result; combining the transformation results to generate the original processing result using a linear decision function; and normalizing the original processing result using a normalized exponential function to obtain the prediction label. The parameter update module is used to determine the comprehensive loss based on the predicted label and the real label, and update the parameters of the optical front end and / or the electronic back end based on the comprehensive loss.

10. A hybrid optoelectronic computing system, characterized in that, include: Optical front end and electronic back end; The optical front end is used to generate C channel scalars in parallel for the target scene, where C is a positive integer greater than 1. The optical front end corresponds to C channels, and each channel scalar represents the physical response of the corresponding channel to the target scene. The optical front end is an optical front end based on a metasurface array, which is divided into Q blocks. The C channels include C channels determined according to the Q blocks and a predetermined optical front end configuration, where Q is a positive integer greater than 1. The optical front end includes the metasurface array and the following components: an image sensor for receiving a light field modulated by an input light field and forming pixel values ​​corresponding to the light intensity distribution through photoelectric conversion and integration; and a sensor readout module for performing the following processing for each channel: using the readout window function corresponding to the channel, weighted summing of the pixel values ​​of the corresponding area of ​​the block corresponding to the channel on the image sensor, and determining the weighted sum as the channel scalar, wherein each block corresponds to a specific area on the image sensor. The electronic backend is used to generate the task processing result corresponding to the target scenario based on the C channel scalars; The electronic backend includes a digital processing module, which performs nonlinear transformations on each channel scalar using the corresponding outer univariate function to obtain transformation results, combines the transformation results to generate original processing results using a linear decision function, and normalizes the original processing results using a normalized exponential function to obtain the task processing result.

11. The system according to claim 10, characterized in that, The optical front end also includes: a polarization analyzer and a hyperspectral selection module; The polarization analyzer is used to implement S different polarization configurations, and the hyperspectral selection module is used to implement L different hyperspectral discrete band configurations; the metasurface array, the polarization analyzer, and the hyperspectral selection module work together to modulate the input light field, where S and L are both positive integers greater than 1.

12. The system according to claim 11, characterized in that, Also includes: Metasurface driving and control module and clock and synchronization control module; The metasurface driving and control module is used to drive the metasurface array with voltage and to control the metasurface array with time division multiplexing according to T time division multiplexing configurations, where T is a positive integer greater than 1. The clock and synchronization control module is used to perform timing synchronization control on the various components in the optical front end.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

15. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method of any one of claims 1-8.

Images