Multi-task diffractive neural network device and processing method based on multi-wavelength parallelism

Abstract

The application relates to the technical field of optical computing and artificial intelligence, and provides a multi-task diffraction neural network device and processing method based on multi-wavelength parallelism, the device comprising an input unit, a diffraction modulation structure, a light detection assembly and a processing unit; the input unit is used for modulating the inputs of N tasks to N wavelengths, inputting a mixed light beam formed after light field superposition into the diffraction modulation structure, and the diffraction modulation structure is used for outputting after parallel processing of each wavelength component of the input mixed light beam; the light detection assembly is used for detecting the light intensity of the output plane of the diffraction modulation structure; the output plane comprises M categories of detection regions, and each detection region comprises N sub-regions; and the processing unit is used for determining the inference results of the N tasks corresponding to the N wavelengths according to the light intensity distribution of the corresponding sub-region in each detection region. The problems that D2NN can only adapt to a single deep learning task and has poor universality are solved, multi-task parallelism is realized, and the universality is improved.

Description

Technical Field

[0001] This invention relates to the fields of optical computing and artificial intelligence, and in particular to a multi-wavelength parallel multi-task diffraction neural network device and processing method. Background Technology

[0002] With the development of science and technology, photonic computing has been widely used due to its advantages such as light-speed processing, low power consumption, and high throughput. For example, photons can be used instead of electrons to perform artificial intelligence (AI) tasks. Currently, photonic neural networks are based on photonic computing to realize artificial neural network models, which can greatly improve computing speed and energy efficiency. Among different photonic neural network architectures, the diffractive deep neural network (D2NN) can perform large-scale neural information processing. D2NN consists of multiple diffractive layers composed of diffractive optical elements. In practical applications, D2NN can only adapt to a single task and has poor versatility. Summary of the Invention

[0003] This invention provides a multi-task diffraction neural network device and processing method based on multi-wavelength parallelism, which solves the shortcomings of existing technologies where D2NN can only adapt to a single task and has poor versatility, and realizes multi-task parallelism, thereby improving versatility.

[0004] This invention provides a multi-wavelength parallel multi-task diffraction neural network device, comprising: an input unit, a diffraction modulation structure, a photodetector component, and a processing unit; wherein, the diffraction modulation structure comprises multiple diffraction layers, and each diffraction layer comprises multiple diffraction optical elements;

[0005] The input unit is used to modulate the inputs of N tasks to N wavelengths, and after the light fields are superimposed to form a mixed beam, it is input into the diffraction modulation structure. The N wavelengths correspond one-to-one with the inputs of the N tasks.

[0006] The diffraction modulation structure is used to process the wavelength components of the input mixed beam in parallel and then output them.

[0007] The photodetector is used to detect the light intensity of the output plane of the diffraction modulation structure; the output plane includes M types of detection regions, each detection region contains N sub-regions, and the N sub-regions correspond one-to-one with the N wavelengths; where M and N are both positive integers.

[0008] The processing unit is used to determine the inference results of N tasks corresponding to N wavelengths based on the light intensity distribution of each sub-region in each detection region.

[0009] According to the present invention, a multi-wavelength parallel multi-task diffraction neural network device is provided, wherein the processing unit is specifically used for:

[0010] For each task, from the sub-regions corresponding to the wavelengths corresponding to the input of the task within the M detection regions, select the sub-region with the highest light intensity, and use the category of the detection region containing the sub-region with the highest light intensity as the inference result of the task.

[0011] According to the present invention, a multi-wavelength parallel multi-task diffraction neural network device is provided, wherein the phase modulation coefficient of each diffraction optical element is obtained in the following manner:

[0012] Based on the error between the detected value and the true value of the light intensity of the sub-region corresponding to the wavelength of the task input, a first loss function is determined;

[0013] The second loss function is determined based on the sum of light intensities outside each of the sub-regions corresponding to the wavelengths corresponding to the input of the task.

[0014] Based on the first loss function and the second loss function, determine the target loss function;

[0015] Based on the target loss function, the phase modulation coefficient of each diffractive optical element is determined.

[0016] According to the present invention, a multi-task diffraction neural network device based on multi-wavelength parallelism is provided, wherein the N tasks come from different datasets.

[0017] According to the present invention, a multi-wavelength parallel multi-task diffraction neural network device is provided, wherein the photodetector component includes a photodetector for detecting the light intensity of the entire output plane; or, the photodetector component includes a photodetector corresponding to each sub-region.

[0018] According to the present invention, a multi-wavelength parallel multi-task diffraction neural network device is provided, wherein the input unit includes N-1 beamsplitters arranged sequentially along the optical path direction; when the beamsplitter is the first beamsplitter in the optical path direction, the input of the beamsplitter includes the input of the task corresponding to two wavelengths; when the beamsplitter is not the first beamsplitter in the optical path direction, the input of the beamsplitter includes the output of the previous beamsplitter and the input of the task corresponding to one wavelength; when the beamsplitter is the last beamsplitter in the optical path direction, the output of the beamsplitter is the mixed beam.

[0019] The present invention also provides a processing method applied to any of the above-described multi-wavelength parallel multi-task diffraction neural network devices, comprising:

[0020] After the input unit modulates the inputs of N tasks to N wavelengths, and the light fields are superimposed to form a mixed beam that is input to the diffraction modulation structure, the photodetector component detects the light intensity of the output plane of the diffraction modulation structure.

[0021] The processing unit determines the inference results of N tasks corresponding to N wavelengths based on the light intensity of each sub-region in each detection region of the output plane.

[0022] According to the processing method of the present invention applied to any of the above-described multi-wavelength parallel multi-task diffraction neural network devices, the processing unit determines the inference results of N tasks based on the light intensity of each sub-region in each detection region of the output plane, including:

[0023] For each task, from the sub-regions corresponding to the wavelengths corresponding to the input of the task within the M detection regions, select the sub-region with the highest light intensity, and use the category of the detection region containing the sub-region with the highest light intensity as the inference result of the task.

[0024] According to the processing method of the present invention applied to any of the above-described multi-wavelength parallel multi-task diffraction neural network devices, the phase modulation coefficients of each of the diffraction optical elements are obtained in the following manner:

[0025] Based on the error between the detected value and the true value of the light intensity of the sub-region corresponding to the wavelength of the task input, a first loss function is determined;

[0026] The second loss function is determined based on the sum of light intensities outside each of the sub-regions corresponding to the wavelengths corresponding to the input of the task.

[0027] Based on the first loss function and the second loss function, determine the target loss function;

[0028] Based on the target loss function, the phase modulation coefficient of each diffractive optical element is determined.

[0029] According to the processing method provided by the present invention, which is applied to any of the above-described multi-wavelength parallel multi-task diffraction neural network devices, the task is an image classification task.

[0030] The present invention provides a multi-wavelength parallel multi-task diffraction neural network device. An input unit modulates the inputs of N tasks onto N wavelengths, which are then superimposed by light fields to form a mixed beam, which is then input into a diffraction modulation structure. The diffraction modulation structure processes each wavelength component of the input mixed beam in parallel and outputs the result. A photodetector detects the light intensity of the output plane of the diffraction modulation structure. This output plane includes M categories of detection regions, each containing N sub-regions. Each of the N sub-regions corresponds one-to-one with one of the N wavelengths. Based on this, the processing unit can determine the inference results of the N tasks corresponding to the N wavelengths according to the light intensity of each sub-region in each detection region, thus achieving parallel processing of multiple tasks and realizing a multi-wavelength D2NN. Compared with existing technologies, multi-wavelength D2NN can encode the inputs of different tasks onto different wavelengths, achieving parallel processing of different tasks. This not only provides strong versatility but also significantly improves computational throughput. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 This is one of the schematic diagrams of the multi-wavelength parallel multi-task diffraction neural network device provided by the present invention;

[0033] Figure 2 This is the second schematic diagram of the multi-wavelength parallel multi-task diffraction neural network device provided by the present invention;

[0034] Figure 3 This is the third schematic diagram of the multi-wavelength parallel multi-task diffraction neural network device provided by the present invention;

[0035] Figure 4 This is the fourth schematic diagram of the multi-wavelength parallel multi-task diffraction neural network device provided by the present invention;

[0036] Figure 5 This is the fifth schematic diagram of the multi-wavelength parallel multi-task diffraction neural network device provided by the present invention;

[0037] Figure 6 This is the sixth schematic diagram of the multi-wavelength parallel multi-task diffraction neural network device provided by the present invention;

[0038] Figure 7 This is a schematic diagram of the processing method provided by the present invention for a multi-task diffraction neural network device based on multi-wavelength parallelism. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0040] In existing technologies, D2NNs cannot reuse different tasks in parallel. The inventors found that realizing the reuse of different tasks in D2NNs is of great significance for improving their versatility and expanding their application in different scenarios. However, executing multiple AI tasks in parallel with D2NNs remains very challenging. During training, competition between different tasks is a major obstacle because it can lead to catastrophic forgetting. Catastrophic forgetting usually occurs during multi-task training. When learning a new task, knowledge of previously learned tasks is suddenly lost, which in turn degrades the performance of each task. In traditional solutions, it is necessary to adjust optical components such as diffraction optical elements in D2NNs one by one to switch between single tasks, or to design multiple different D2NNs to match different tasks, which greatly increases hardware complexity.

[0041] Therefore, this invention provides a multi-task diffraction neural network device based on multi-wavelength parallelism, which can implement multi-task in D2NN. The following is in conjunction with... Figures 1-6 The present invention describes a multi-wavelength parallel multi-task diffraction neural network device.

[0042] This embodiment provides a multi-wavelength parallel multi-task diffraction neural network device, such as... Figure 1 As shown, it includes: an input unit 110, a diffraction modulation structure 120, a photodetector assembly 130, and a processing unit 140; wherein, the diffraction modulation structure 120 includes multiple diffraction layers, and each diffraction layer includes multiple diffraction optical elements;

[0043] The input unit 110 is used to modulate the inputs of N tasks to N wavelengths, and after the light fields are superimposed to form a mixed beam, it is input into the diffraction modulation structure. The N wavelengths correspond one-to-one with the inputs of the N tasks.

[0044] The diffraction modulation structure 120 is used to process and output the wavelength components of the input mixed beam in parallel.

[0045] The photodetector 130 is used to detect the light intensity of the output plane of the diffraction modulation structure; the output plane includes M types of detection regions, each detection region contains N sub-regions, and the N sub-regions correspond one-to-one with the N wavelengths; where M and N are both positive integers.

[0046] The processing unit 140 is used to determine the inference results of N tasks corresponding to N wavelengths based on the light intensity distribution of each sub-region in each detection region.

[0047] In this embodiment, multiple diffractive optical elements are densely arranged in the diffraction layer. The input unit 110, diffraction modulation structure 120, photodetector assembly 130, and processing unit 140 form a D2NN.

[0048] In this model, M and N can be greater than or equal to 2, allowing for the simultaneous processing of multiple tasks. The N tasks can originate from the same dataset, or they can originate from different datasets, enabling parallel completion of multiple tasks—a feature known as optical multitasking. Simultaneously, the inputs of the N tasks are modulated onto N wavelengths, and after superposition of the light fields to form a mixed beam, this beam is input into a diffraction modulation structure. The diffraction modulation structure processes the wavelength components of the input mixed beam in parallel and outputs the results. Subsequent photodetector components and processing units then work together to obtain the inference results for each wavelength-carried task. Based on this, the corresponding D2NN can be called a multi-wavelength D2NN. Here, the wavelength dimension is utilized to improve computational throughput by encoding (i.e., modulating) different inputs onto different wavelengths and performing photonic computations in both spatial and spectral dimensions. This achieves highly parallel processing of multiple inputs.

[0049] The tasks can be image processing tasks, such as image classification, or other image processing tasks. Correspondingly, the inference result of the task is the image classification result. N tasks include N images, which can be images acquired under different wavelengths of light, and the images can contain the target object to be classified, etc.

[0050] Thus, multi-wavelength D2NNs inherently possess the advantages of parallel processing of multiple tasks, offering high processing speed, low power consumption, and high throughput. By encoding different tasks onto different wavelengths, competition between different tasks can be significantly alleviated while maintaining high performance for each task. For each new task, new wavelengths can be easily added to implement the new task at extremely low cost. Therefore, multi-wavelength D2NNs can fully utilize photonic computing, facilitating the realization of more general brain-like intelligent architectures. In implementation, multiple tasks can be executed simultaneously on different datasets without the need for adjustments between tasks.

[0051] In implementation, there can be M categories. Detection regions for each category can be set on the output plane of the D2NN, i.e., detection regions for the M categories. Furthermore, the detection region for each category is divided into N parts, resulting in N sub-regions. The N sub-regions correspond one-to-one with the N wavelengths, where each sub-region represents the category of the input at the corresponding wavelength. Figure 2 The diagram is illustrated with 9 categories, corresponding to detection areas 0, 1, 2, ..., 9. Each detection area has multiple different sub-areas.

[0052] like Figure 2 As shown, in a preset order, among N wavelengths, the i-th task corresponds to the i-th wavelength λ. i wavelength λ i Encode the information for the i-th task, where i = 1, ..., N. Figure 2 The diagram illustrates a diffraction layer configuration of 5 layers, denoted as L1, L1, ... L5. The mixed beam can be input into the diffraction modulation structure via an aperture stop.

[0053] Based on the approximation theory of multi-wavelength optical systems, the transformation of a multi-wavelength optical field can be viewed as a combination of independent transformations of coherent optical fields of each wavelength, following the principle of superposition of light intensity. Therefore, the input optical field with wavelength λi encodes the i-th task and is detected after propagating through the diffraction modulation structure. In the implementation, a linear D2NN with a complex transform function M(Φ) is considered, where Φ represents the phase modulation coefficient of the diffractive optical elements in multiple pure phase diffraction layers. The complex transform function can be found in relevant techniques and will not be elaborated upon here.

[0054] In this embodiment, based on the design of multi-wavelength diffractive optical elements (DOEs), the phase modulation coefficient of each diffraction layer is the same at different wavelengths. Therefore, the wavelength λ i Output light field U i This can be expressed as:

[0055] U i `=M i (Φ)U i (1)

[0056] Among them, U i Indicates wavelength λ i Given the input light field, the corresponding output light field intensity distribution of the diffraction modulation structure can be expressed as:

[0057] I i =|U i `| 2 =|M i (Φ)U i | 2 (2)

[0058] I i Indicates wavelength λ i For a multi-wavelength D2NN, the total light field intensity distribution I at different wavelengths can be expressed as the superposition of the light field intensity distributions detected at the sub-regions corresponding to each wavelength:

[0059] I = Σ i I i =Σ i |M i (Φ)U i | 2 (3)

[0060] Correspondingly, in the M categories of detection regions on the output plane, according to a preset order, the j-th detection region includes N sub-regions, j = 1, ..., M, which are indices of different detection regions and can represent different categories. The i-th sub-region of the j-th detection region is denoted as D. j i Where i = 1, ..., N, represents the wavelength λ i The index of the corresponding task.

[0061] In practice, the wavelength λ can be selected using an optical filter. i Then, the light intensity is detected by a photodetector. An optical filter, also known as a wavelength-selective filter, is used for wavelength selection. A wavelength-selective filter can be applied to each sub-region to eliminate crosstalk between wavelength channels during intensity detection, improving task performance. In this case, each sub-region only detects the light intensity at its corresponding wavelength, i.e., sub-region D. j i Light intensity I(D) j i ) = I i (D j i ), that is, subregion D j i wavelength λ i The light intensity on the surface. Alternatively, without a wavelength-selective filter, sub-region D... j i The light intensity is the sum of the light intensities of all wavelength components.

[0062] Compared to spatial multiplexing in multiple D2NNs, in multi-wavelength diffraction computation, the light signals of different wavelengths are independent of each other and there is no crosstalk. Therefore, increasing the number of wavelengths and diffraction optical elements in a D2NN can increase computational throughput and simplify the processing of more tasks.

[0063] Thus, in this embodiment, the inputs of N tasks are modulated to N wavelengths by the input unit, and after being superimposed by the light fields to form a mixed beam, it is input to the diffraction modulation structure. The diffraction modulation structure is used to process the wavelength components of the input mixed beam in parallel and output them. The light intensity of the output plane of the diffraction modulation structure is detected by the photodetector component. The output plane includes M types of detection areas, each containing N sub-regions. The N sub-regions correspond one-to-one with the N wavelengths. Based on this, the processing unit can determine the inference results of the N tasks corresponding to the N wavelengths according to the light intensity of each sub-region in each detection area, thereby realizing the parallel processing of multiple tasks and realizing multi-wavelength D2NN. In this way, compared with the prior art, the inputs of different tasks can be encoded to different wavelengths through multi-wavelength D2NN to realize the parallel processing of different tasks. It is not only highly versatile, but also greatly improves the computational throughput.

[0064] In an exemplary embodiment, the input unit includes N-1 beamsplitters arranged sequentially along the optical path direction; when the beamsplitter is the first beamsplitter in the optical path direction, the input of the beamsplitter includes the input of the task corresponding to two wavelengths; when the beamsplitter is not the first beamsplitter in the optical path direction, the input of the beamsplitter includes the output of the previous beamsplitter and the input of the task corresponding to one wavelength; when the beamsplitter is the last beamsplitter in the optical path direction, the output of the beamsplitter is the mixed beam.

[0065] The beam splitter is a half-reflective half-lens. For example... Figure 3 As shown, N wavelengths λ1, λ2, λ3, ..., λ N There are N-1 beamsplitters arranged sequentially along the optical path, as indicated by the thin arrows at the bottom of the diagram. Along the optical path, the first beamsplitter receives inputs from two wavelengths corresponding to the target wavelengths: one transmitted light and the other reflected light. Starting with the second beamsplitter, the inputs include the output of the previous beamsplitter (transmitted light) and the input of one wavelength corresponding to the target wavelength (reflected light). This continues until the last beamsplitter, which mixes all the target wavelength inputs to obtain a mixed beam, which is then output to the diffraction modulation structure. In this embodiment, modulation of inputs from different targets can be achieved using multiple beams, resulting in a simple and easily implemented structure.

[0066] In an exemplary embodiment, the processing unit is specifically used for:

[0067] For each task, from the sub-regions corresponding to the wavelengths corresponding to the input of the task within the M detection regions, select the sub-region with the highest light intensity, and use the category of the detection region containing the sub-region with the highest light intensity as the inference result of the task.

[0068] The light detection component includes a photodetector for detecting the light intensity of the entire output plane; or, the light detection component includes a photodetector corresponding to each sub-region. The photodetector can be a grayscale camera, a light intensity sensor, etc. In implementation, appropriate settings can be flexibly selected according to actual needs.

[0069] Specifically, determine the wavelength λ i Corresponding subregion D 1 i D 2 i , ...,D M i The light intensity of each sub-region is obtained, and the category of the detection region containing the sub-region with the highest light intensity is determined by wavelength λ. i The corresponding task category.

[0070] In this embodiment, for a certain task, the greater the light intensity of each sub-region of the detection area, the higher the probability of belonging to the category of that detection area. Therefore, taking the category of the detection area where the sub-region with the greatest light intensity is located as the category of the task can achieve more accurate classification.

[0071] In an exemplary embodiment, the phase modulation coefficients of each of the diffractive optical elements are obtained in the following manner:

[0072] Based on the error between the detected value and the true value of the light intensity of the sub-region corresponding to the wavelength of the task input, a first loss function is determined;

[0073] The second loss function is determined based on the sum of light intensities outside each of the sub-regions corresponding to the wavelengths corresponding to the input of the task.

[0074] Based on the first loss function and the second loss function, determine the target loss function;

[0075] Based on the target loss function, the phase modulation coefficient of each diffractive optical element is determined.

[0076] In practical applications, multi-wavelength D2NNs can be pre-trained to achieve optical multi-task learning. For example, a joint optimization method can be used to train a multi-wavelength D2NN. This joint optimization method employs the training method described above using the objective loss function.

[0077] The target loss function comprises two parts. One part is the first loss function mentioned above, which is obtained based on the error between the detected value and the true value of the light intensity of the sub-region corresponding to the wavelength of the task. The target loss function aims to minimize the error between the detected value and the true value, which can improve the classification accuracy of D2NN. The other part is the second loss function mentioned above, which is obtained based on the sum of the light intensity outside each sub-region corresponding to the wavelength of the task. The target loss function also aims to minimize the sum of the light intensity outside the detection area to maximize the energy transmission efficiency of multi-wavelength D2NN, which can further improve the classification accuracy of D2NN.

[0078] For example, the target loss function includes:

[0079]

[0080] Where L(Gi, Pi) represents the execution wavelength λ i In the corresponding task, the softmax cross-entropy loss function, i.e., the first loss function, is generated by the error between the detected value Pi and the true value Gi of the light intensity in the sub-region, and is called the softmax cross-entropy term. Here, Gi is a one-hot vector of length M. The sum of light intensity outside the sub-region of the detection area, as evaluated by mean square error, is represented by the second loss function mentioned above, and is called the energy efficiency constraint term, where I i I i That is, wavelength λ i Total light intensity, That is, wavelength λ i The light intensity of each sub-region of the corresponding detection area The sum of these, MSE() represents the mean squared error.

[0081] During training, different wavelengths share the same phase modulation coefficient in each diffraction layer. Optical multitasking can be achieved by iteratively updating the optimal solution of equation (4).

[0082] In practice, a stochastic gradient descent method can be used to train a multi-wavelength D2NN. The inputs of training datasets for different tasks are encoded as light field amplitudes of different wavelengths and fed into the D2NN. Error backpropagation is performed according to the target loss function to optimize the network structure of the D2NN and the phase modulation coefficients of the diffractive optical elements.

[0083] In the fabrication of multi-wavelength diffractive optical elements, it is possible to achieve the same phase modulation characteristics across different wavelengths. For example, the geometry of each diffractive optical element can be determined such that the optical path length of each element has the same phase value for each wavelength. Specifically, this can be achieved by adding an integral multiphase delay (e.g., 2π) at one wavelength until an appropriate phase delay is reached at another wavelength. The overall physical height will be determined based on the actual precision required. Alternatively, the optical path length for each wavelength can be controlled by utilizing the refractive index variation of the dispersive material at different wavelengths. It can also be designed by combining several aligned diffractive optical elements made of different materials, similar to polarization-selective diffractive optical elements. The flexibility of wavefront operation across different physical dimensions (e.g., phase, amplitude, wavelength, and polarization) makes it possible for diffractive optical elements to encode multiple wavelengths. For example, a subsurface composed of different types of nanobulbs, whose spatially varying rotation angles are multiplexed within a subwavelength unit, can enable the diffractive optical element to resonate with different wavelengths.

[0084] The solution provided by this invention will be described in more detail below through specific scenarios.

[0085] In this embodiment, an image classification task is used as an example. First, the application of multi-wavelength D2NN in highly parallel classification tasks is verified, demonstrating that it can classify multiple inputs simultaneously while performing a single classification task.

[0086] Specifically, a three-wavelength D2NN with five diffraction layers was built using the PyTorch deep learning framework to classify the MNIST dataset, which can recognize three handwritten digits at each time step. Considering visible wavelengths from 400nm to 700nm, the input light source was set to a combination of three wavelengths: 400nm, 550nm, and 700nm, each encoding one of the three handwritten digits. Therefore, the detection region for each class on the output plane was correspondingly segmented into three sub-regions.

[0087] The Adam optimizer can be used for D2NN training to optimize the phase modulation coefficients of the diffractive optical elements. The size of each diffractive optical element is set to 4µm × 4µm.

[0088] First, the performance of the multi-wavelength D2NN was evaluated by setting the number of diffractive optical elements in each diffraction layer to 200×200, corresponding to a diffraction layer size of 0.8mm×0.8mm (see [reference]). Figure 4 Further evaluation and comparison of network performance under different numbers of diffractive optical elements in each layer, i.e., K×K, K=200, 400, 600, 800 (see...). Figure 4The number of diffraction layers was set to 5, and the distance between consecutive layers was optimized according to diffraction theory. The training batch size was set to 32, the initial learning rate was set to 0.01, and halved (multiplied by 0.5) after each epoch during training. D2NN training converged after five epochs to achieve an ideal mapping function for multi-wavelength inputs and outputs. D2NN was trained with 60,000 handwritten digits and blind-tested with 10,000 handwritten digits. For a diffraction layer with K×K diffractive optical elements, each digit with a pixel number of 28×28 was first adjusted to K / 2×K / 2 and then padded to K×K.

[0089] Numerical evaluation results are as follows Figure 4 Parts b and c are shown, in which the performance of multi-wavelength D2NN is effective in the detection region of each class, regardless of whether a wavelength-selective filter is used. Figure 4 The ac section shows example results of simultaneously classifying three handwritten input digits: "7", "2", and "5", encoded at wavelengths of 700nm, 550nm, and 400nm respectively. Each layer contains 200×200 diffractive optical elements. The classification result for each wavelength is determined by the maximum light intensity between the sub-regions corresponding to the detection area of ​​the category. The three input digits are represented by three white arrows, as shown below. Figure 4 The left side of sections b and c is shown. Figure 4 The energy distribution of the classification results of the three inputs in parts b and c at different wavelengths shows that D2NN can significantly identify the sub-region with the maximum light intensity and thus classify it correctly.

[0090] Since wavelength selective filters can eliminate wavelength crosstalk during the detection process, the classification accuracy of multi-wavelength D2NN using wavelength selective filters is 95.9%, 96.4%, and 96.9% for wavelengths of 700nm, 550nm, and 400nm, respectively. This is slightly higher than the classification accuracy of broadband wavelength detection without wavelength selective filters, which is 95.0%, 95.7%, and 96.4%, respectively.

[0091] For both setups, as the number of diffractive optical elements per diffraction layer increases, the classification accuracy of multi-wavelength D2NN at each wavelength further improves, such as... Figure 4As shown in section d, with 800×800 diffractive optical elements per layer, the classification accuracy of the multi-wavelength D2NN with wavelength-selective filters reaches 98.2%, 98.1%, and 98.1% for wavelengths of 700nm, 550nm, and 400nm, respectively. This is equivalent to training three single-wavelength D2NNs with serial input, i.e., sequential digital input. The results show that multi-wavelength D2NN can significantly improve parallel computing capabilities. By encoding different classification tasks onto different wavelengths, multi-task learning using multi-wavelength D2NN allows for the parallel execution of different machine learning tasks within the D2NN.

[0092] Regarding the ability of multi-wavelength D2NN for optical multi-task learning, a dual-task classifier was first constructed to classify the MNIST and Fashion-MNIST (FMNIST) datasets. The classification task for the MNIST dataset is denoted as Task I, and the classification task for the FMNIST dataset is denoted as Task II. Both datasets include 60,000 training samples and 10,000 test samples, with 10 class numbers and correspondingly 10 detection regions. Therefore, the dual-wavelength D2NN is constructed by dividing each of the ten detection regions into two sub-regions. See [link to documentation]. Figure 5 The section below (a) shows the classification results for Task I and Task II, respectively. Task I used a 700nm wavelength for handwritten digit encoding, while Task II used a 400nm wavelength for fashion product encoding. Other network settings and... Figure 4 Under the same conditions, the two wavelength D2NNs are first set to have five diffraction layers, with each diffraction layer having 200×200 diffraction optical elements, and the photodetector does not contain a wavelength selection filter. Figure 5 Section a shows example results of classifying the handwritten digit "7" (class number 7) in the MNIST dataset and the fashion product "pullover" (class number 2) in the FMNIST dataset. Figure 5 In Part a, the energy distribution of the classification results for the two tasks shows that multi-wavelength D2NN can significantly identify the sub-regions with the highest light intensity, thus achieving correct classification. The maximum light intensity outputs for Task I and Task II are concentrated in the upper sub-region of detection region 7 and the lower sub-region of detection region 2, respectively.

[0093] Blind testing of the trained dual-wavelength D2NN on the MNIST and FMNIST test datasets yielded classification accuracies of 95.6% and 86.8%, respectively. Figure 5 Sections b and c show the corresponding confusion matrix and energy distribution matrix, respectively, and statistically summarize the classification results of all samples and the energy distribution percentages for the two tasks. The average energy percentages for the correct classes on the two tasks are 20.8% and 21.8%, respectively.

[0094] Furthermore, the performance of dual-wavelength D2NN for parallel execution of two tasks was compared with that of single-wavelength D2NN for parallel execution of two tasks by overlapping and reusing two images from two datasets as input, as shown in Table 1. Single-wavelength D2NN achieved parallel execution rates of 92.4% and 83.1% for the two tasks, respectively, which is significantly lower than that of dual-wavelength D2NN.

[0095] In addition, two single-wavelength D2NNs were trained for each of the two tasks, achieving classification accuracies of 97.1% and 87.5% for Task I and Task II, respectively. To improve the performance of the dual-wavelength D2NN, the number of diffraction elements per diffraction layer was increased. For the two tasks with 400×400 diffraction elements per diffraction layer, classification accuracies reached 97.5% and 88.0%, respectively. Using wavelength-selective filters in the detection region of the categories further improved performance. For the two tasks with 200×200 diffraction elements per diffraction layer, classification accuracies reached 95.9% and 87.0%, respectively; for 400×400 diffraction elements per diffraction layer, classification accuracies reached 97.6% and 88.9%, respectively. These results showed comparable or even higher classification accuracies compared to training two single-wavelength D2NNs separately to perform the two tasks. Table 1 summarizes the experimental results, verifying that the designed dual-wavelength D2NN with joint optimization method can successfully classify targets in parallel from two tasks without any adjustment to the diffraction layers of the two tasks.

[0096] Table 1 Experimental Results

[0097]

[0098]

[0099] Regarding the capabilities of multi-wavelength D2NN for optical multi-task learning, a four-wavelength D2NN was constructed for four task classifications. It can simultaneously classify data from the MNIST, FMNIST, Kuzushiji MNIST (KMNIST), and Extended MNIST (EMNIST) datasets, designated Task I, Task II, Task III, and Task IV, respectively. KMNIST contains images of ancient characters with the same dataset size and class numbers as the MNIST and FMNIST datasets. Ten classes of handwritten letters were randomly selected from the EMNIST dataset, maintaining the same dataset size as the other three tasks: 60,000 training samples and 10,000 test samples. The datasets for the four tasks (Task I to Task IV) are encoded with wavelengths of 700nm, 600nm, 500nm, and 400nm, respectively. In this numerical experiment, the designed four-wavelength D2NN did not use wavelength selection filters, which have lower hardware complexity. Other network settings and... Figure 4 and Figure 5 Under the same conditions, the classification accuracy of a four-wavelength D2NN was evaluated when executing four tasks in parallel at different network sizes, and the classification accuracy was compared with that of a single-wavelength D2NN. Figure 6 As shown, for a four-wavelength D2NN with five diffraction layers and 200 × 200 diffractive optical elements per layer, the classification accuracy for the four tasks, from task I to task IV, was 92.8%, 83.0%, 81.0%, and 90.4%, respectively, significantly higher than the 64.6%, 68.7%, 52.5%, and 55.3% of single-wavelength D2NNs with the same network size. At different network sizes, the four-wavelength D2NN consistently achieved higher accuracy than the single-wavelength D2NN for four-task classification. As the number of tasks increased from two to four, multi-wavelength D2NNs showed even greater advantages in achieving multi-task optical learning.

[0100] Furthermore, the performance of the four-wavelength D2NN on the four tasks at different network sizes was evaluated and compared with the individual training performance of the four single-wavelength D2NNs. See [link to documentation]. Figure 6 Part a increases the network size by increasing the number of diffraction layers from 1 to 8, with each diffraction layer containing 200 × 200 diffraction optical elements. Figure 6Part b increases the network size by increasing the number of diffractive optical elements in each diffraction layer with the same number of layers (5). Increasing the neural network size of the multi-wavelength D2NN used for optical multi-task learning significantly improves its inference ability until performance saturates. The performance of the four-wavelength D2NN improves with increasing network size, approaching the performance of training four single-wavelength D2NNs. The performance rates for Task I and Task IV are 96.5%, 85.6%, 88.6%, and 93.8%, respectively, with 5 diffraction layers and 800×800 diffractive optical elements per layer. Under this configuration, the four-wavelength D2NN exhibits comparable performance to four single-wavelength D2NNs of the same network size. The results demonstrate that our proposed method is effective for multi-task learning of D2NNs and achieves lower hardware complexity. Encoding the input of multi-classification tasks into multiple wavelengths effectively alleviates competition between different classification tasks and minimizes the performance degradation of each classification task.

[0101] The multi-wavelength D2NN in this embodiment is an all-optical computing processor that can execute multiple classification tasks simultaneously with extremely low computational latency. The total computational time latency for each instance of the multi-classification task input is the sum of the wavefront propagation time from the input plane to the photodetector plane and the response time of the photodetector, independent of the number of wavelength channels and diffractive optical elements in each layer. Figure 6 Taking a four-wavelength D2NN with five diffraction layers and 800×800 diffractive optical elements per layer as an example, assuming a detection rate of 30 GHz, the total computation time per instance of the multi-classification task input is 1.23 ns, resulting in approximately 324 million diffraction optical computations. Furthermore, compared to spatial multiplexing in multiple D2NNs, in multi-wavelength diffraction optical computation, the light signals of different wavelengths are independent of each other, with no crosstalk. Therefore, increasing the number of wavelengths and diffractive optical elements in a D2NN system can increase computational throughput and simplify more classification tasks.

[0102] For each new classification task, new wavelengths can be easily added to achieve the new classification task. The classification task expansion process is as follows: Figure 6 As shown in part a, the cost of the entire process is extremely low.

[0103] The above demonstrates the high parallelism of three-wavelength D2NN by performing parallel classification on three different inputs using the MNIST dataset, where the accuracy of each wavelength is equivalent to training three single-wavelength D2NNs with sequential inputs. To perform multiple classification tasks in parallel, inputs from different datasets are encoded into different wavelengths. Two-wavelength and four-wavelength D2NNs were used to perform two-task and four-task classification on the MNIST, FMNIST, KMNIST, and EMNIST datasets, respectively. As the number of tasks increases, multi-wavelength D2NN achieves higher classification accuracy than single-wavelength D2NN and maintains the classification accuracy of the model for each task even with larger network sizes. This sufficiently demonstrates the significant advantages of multi-wavelength D2NN in achieving optical multi-task learning.

[0104] It should be noted that, Figure 4 The classification accuracy is represented by a dot for single-task single-wavelength, a box for single-task multi-wavelength (including wavelength selection filter), and a triangle for single-task multi-wavelength (excluding wavelength selection filter).

[0105] and Figure 6 The classification accuracy is represented by a dot for single-task single-wavelength, a box for multi-task multi-wavelength, and a triangle for multi-task single-wavelength.

[0106] Thus, by encoding multi-classification tasks into multiple wavelengths to utilize the wavelength dimension of the diffracted light field, different tasks can be implemented in parallel at the speed of light through optical multi-task learning methods. The optical multi-task functionality is implemented in a D2NN, eliminating the need for mechanical movement of the diffraction layer, thereby significantly reducing system complexity. Analysis shows that this method can significantly alleviate competition between multiple tasks while maintaining the performance of each classification task. As the number of classification tasks increases, multi-wavelength D2NNs demonstrate greater advantages in implementing optical multi-task learning. By using wavelength division multiplexing (WDM) techniques to perform optical multi-task learning, this method can be extended to other photonic neural network architectures, achieving high parallelism, high accuracy, and high versatility.

[0107] The following describes the processing method provided by the present invention for a multi-task diffraction neural network device based on multi-wavelength parallelism as provided in any of the above embodiments. This method can be referred to in correspondence with the multi-task diffraction neural network device based on multi-wavelength parallelism described above.

[0108] This embodiment provides a processing method for multi-wavelength parallel multi-task diffraction neural network devices as provided in any of the above embodiments, such as... Figure 7 As shown, it includes:

[0109] Step 701: After the input unit modulates the inputs of N tasks to N wavelengths and forms a mixed beam by superimposing the light fields, the light detection component detects the light intensity of the output plane of the diffraction modulation structure.

[0110] Step 702: The processing unit determines the inference results of N tasks corresponding to N wavelengths based on the light intensity of each sub-region in each detection region of the output plane.

[0111] In an exemplary embodiment, the processing unit determines N inference results for the task based on the light intensity of each sub-region in each detection region of the output plane, including:

[0112] For each task, from the sub-regions corresponding to the wavelengths corresponding to the input of the task within the M detection regions, select the sub-region with the highest light intensity, and use the category of the detection region containing the sub-region with the highest light intensity as the inference result of the task.

[0113] In an exemplary embodiment, the phase modulation coefficients of each of the diffractive optical elements are obtained in the following manner:

[0114] Based on the error between the detected value and the true value of the light intensity of the sub-region corresponding to the wavelength of the task input, a first loss function is determined;

[0115] The second loss function is determined based on the sum of light intensities outside each of the sub-regions corresponding to the wavelengths corresponding to the input of the task.

[0116] Based on the first loss function and the second loss function, determine the target loss function;

[0117] Based on the target loss function, the phase modulation coefficient of each diffractive optical element is determined.

[0118] In an exemplary embodiment, the task is an image classification task.

[0119] In an exemplary embodiment, the N images come from different datasets.

[0120] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-wavelength parallel multi-task diffraction neural network device, characterized in that, include: The system comprises an input unit, a diffraction modulation structure, a photodetector assembly, and a processing unit; wherein the diffraction modulation structure includes multiple diffraction layers, and each diffraction layer includes multiple diffraction optical elements. The input unit is used to modulate the inputs of N tasks to N wavelengths, and after the light fields are superimposed to form a mixed beam, it is input into the diffraction modulation structure. The N wavelengths correspond one-to-one with the inputs of the N tasks. The task is an image classification task. The diffraction modulation structure is used to process the wavelength components of the input mixed beam in parallel and then output them. The photodetector is used to detect the light intensity of the output plane of the diffraction modulation structure; the output plane includes M types of detection regions, each detection region contains N sub-regions, and the N sub-regions correspond one-to-one with the N wavelengths; where M and N are both positive integers. The processing unit is used to determine the inference results of N tasks corresponding to N wavelengths based on the light intensity distribution of each sub-region in each detection region.

2. The multi-wavelength parallel multi-task diffraction neural network device according to claim 1, characterized in that, The processing unit is specifically used for: For each task, from the sub-regions corresponding to the wavelengths corresponding to the input of the task within the M detection regions, select the sub-region with the highest light intensity, and use the category of the detection region containing the sub-region with the highest light intensity as the inference result of the task.

3. The multi-wavelength parallel multi-task diffraction neural network device according to claim 1, characterized in that, The phase modulation coefficients of each of the diffractive optical elements are obtained in the following manner: Based on the error between the detected value and the true value of the light intensity of the sub-region corresponding to the wavelength of the task input, a first loss function is determined; The second loss function is determined based on the sum of light intensities outside each of the sub-regions corresponding to the wavelengths corresponding to the input of the task. Based on the first loss function and the second loss function, determine the target loss function; Based on the target loss function, the phase modulation coefficient of each diffractive optical element is determined.

4. The multi-wavelength parallel multi-task diffraction neural network device according to any one of claims 1 to 3, characterized in that, The N tasks come from different datasets.

5. The multi-wavelength parallel multi-task diffraction neural network device according to any one of claims 1 to 3, characterized in that, The light detection assembly includes a light detector for detecting the light intensity of the entire output plane; or, the light detection assembly includes a light detector corresponding to each of the sub-regions.

6. The multi-wavelength parallel multi-task diffraction neural network device according to any one of claims 1 to 3, characterized in that, The input unit includes N-1 beamsplitters arranged sequentially along the optical path direction; when the beamsplitter is the first beamsplitter in the optical path direction, the input of the beamsplitter includes the input of the task corresponding to two wavelengths; when the beamsplitter is not the first beamsplitter in the optical path direction, the input of the beamsplitter includes the output of the previous beamsplitter and the input of the task corresponding to one wavelength; when the beamsplitter is the last beamsplitter in the optical path direction, the output of the beamsplitter is the mixed beam.

7. A processing method applied to a multi-wavelength parallel multi-task diffraction neural network device as described in any one of claims 1 to 6, characterized in that, include: After the input unit modulates the inputs of N tasks to N wavelengths, and the light fields are superimposed to form a mixed beam that is input to the diffraction modulation structure, the photodetector component detects the light intensity of the output plane of the diffraction modulation structure. The processing unit determines the inference results of N tasks corresponding to N wavelengths based on the light intensity of each sub-region in each detection region of the output plane.

8. The processing method according to claim 7, characterized in that, The processing unit determines N inference results for the task based on the light intensity of each sub-region in each detection region of the output plane, including: For each task, from the sub-regions corresponding to the wavelengths corresponding to the input of the task within the M detection regions, select the sub-region with the highest light intensity, and use the category of the detection region containing the sub-region with the highest light intensity as the inference result of the task.

9. The processing method according to claim 7, characterized in that, The phase modulation coefficients of each of the diffractive optical elements are obtained in the following manner: Based on the error between the detected value and the true value of the light intensity of the sub-region corresponding to the wavelength of the task input, a first loss function is determined; The second loss function is determined based on the sum of light intensities outside each of the sub-regions corresponding to the wavelengths corresponding to the input of the task. Based on the first loss function and the second loss function, determine the target loss function; Based on the target loss function, the phase modulation coefficient of each diffractive optical element is determined.

Images