A rotatable modulation layer optimization method and system based on a diffraction neural network

Abstract

The present application relates to the cross field of optical computing and artificial intelligence and discloses a rotatable modulation layer optimization method and system based on a diffraction neural network, comprising: S1, loading a target image as an input layer through a spatial light modulator, S2, using a coherent light source to irradiate the input layer; S3, for different tasks, jointly optimizing and training the phase of a multi-layer general diffraction modulation layer and a rotating diffraction modulation layer to obtain the corresponding phase parameters I and phase parameters II of each general diffraction modulation layer and rotating diffraction modulation layer after training; S4, printing the general solidified phase plate of each general diffraction modulation layer and the rotating solidified phase plate of the rotating diffraction modulation layer based on the phase parameters I and phase parameters II obtained in S3, for constructing a physically rotatable diffraction optical network when different tasks are formed. Through the rotating diffraction modulation layer, the present application can realize optical computing tasks in different complex scenes, without the need to retrain the diffraction network, and has the characteristics of multi-task reuse.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of optical computing and artificial intelligence, in particular to a rotatable modulation layer optimization method and system based on a diffractive neural network. BACKGROUND

[0002] With the rapid development of information technology, traditional electronic computing systems face problems such as high power consumption, speed bottleneck and thermal effect when processing massive data and complex computing tasks. To overcome these challenges, optical computing as a new computing paradigm has gradually attracted widespread attention. Optical computing utilizes the parallel propagation, low loss and high bandwidth characteristics of photons, enabling high-speed, low-energy information processing. Since the 1960s, the field of optical computing has evolved from basic optical devices to integrated photonic circuits. In recent years, with the integration of nanofabrication technology and artificial intelligence, optical neural networks have become the core direction of optical computing. Among them, diffractive optical neural networks use the principle of light diffraction to design multi-layer modulation structures, achieving image classification, pattern recognition and other tasks, and showing great potential in edge computing and real-time processing applications.

[0003] However, existing diffractive optical neural networks have significant limitations in practical applications. First, their flexibility is insufficient: the modulation layers of traditional networks usually adopt fixed designs, and once optimized, it is difficult to adapt to dynamic changes in the environment or diverse task requirements. This leads to the need to redesign and manufacture hardware when the network faces diverse application scenarios, increasing the cost and time overhead. Second, existing diffractive networks are often optimized for specific computing tasks, such as single image classification or signal processing, and cannot support multiple task switching at the same time. This single-task-oriented design not only limits the system's versatility and scalability, but also causes resource waste in multi-functional integrated systems, failing to meet the needs of modern computing for adaptability and multi-task processing. SUMMARY

[0004] An object of the present application is to solve at least the above problems and / or deficiencies, and to provide at least the advantages described later.

[0005] To achieve these objects and other advantages of the present application, a rotatable modulation layer optimization method based on a diffractive neural network is provided, comprising:

[0006] S1, loading a target image as an input layer of a diffractive optical network through a phase-type spatial light modulator;

[0007] S2, using a coherent light source to irradiate the input layer, making the loaded phase target forward into the diffractive optical network;

[0008] S3, for different tasks, phase joint optimization training is performed on the multi-layer general diffractive modulation layers and the rotating diffractive modulation layers in the diffractive optical network to obtain phase parameters I and phase parameters II corresponding to the trained general diffractive modulation layers and the rotating diffractive modulation layers;

[0009] S4, after the training, the general solidified phase plates of the general diffractive modulation layers and the rotating solidified phase plates of the rotating diffractive modulation layers are printed based on the phase parameters I and the phase parameters II obtained in S3, and are used to construct the physically rotatable diffractive optical network in different tasks.

[0010] Preferably, in S4, the general diffractive modulation layers and the rotating diffractive modulation layers are used to load the trained modulation phase, each general diffractive modulation layer and rotating diffractive modulation layer is composed of an array of 200*200 independently modulatable neurons, and the size of each neuron is 8 microns, and the phase value is constrained between 0 and 2π by a Sigmoid function.

[0011] Preferably, in S1, the target image is a gray-scale handwritten digit, and the target image is constrained to a phase range of 0 to 2π by using a Sigmoid normalization function.

[0012] Preferably, in S2, a 532nm coherent light source is used to illuminate the input layer, so that the light field loaded with the phase target follows the Rayleigh-Sommerfeld theorem and is transmitted forward into the diffractive optical network, and the propagation form of the light field is characterized by the following formula:

[0013] ;

[0014] In the above formula, represents a pulse response function, represents a complex light field of the i-th diffractive modulation layer at a spatial position r, represents a complex light field of the i+1-th diffractive modulation layer at a spatial position r, represents a coordinate of the i-th diffractive modulation layer at a spatial position r, represents a coordinate of the i+1-th diffractive modulation layer at a spatial position r. r r

[0015] Preferably, in S3, a cross-entropy loss function is used to calculate the loss value of the output vector and the One-hot encoding, and the phase parameters of the diffractive modulation layers are updated by a back propagation algorithm, and the cross-entropy loss function is characterized by the following formula:​​​​​​​​

[0016] L=L ce +L leak

[0017] In the above formula, L ce represents the main classification loss function, and

[0018] In the above formula, is the weight coefficient of task t, and e is One-hot encoding, represents the task t The phase on the i-th diffraction modulation layer, B represents the batch size, represents the probability of the mechanical rotation angle corresponding to task t in the i-th batch; represents the encoding of the y coordinate of task t in the i-th batch, CE() represents the loss function, and i=1, 2…B;

[0019] L leak represents the leakage loss, and

[0020]

[0021] In the above formula, β 1, β 2 are weight coefficients, is the task t The light intensity at the spatial position r, r is the spatial position, represents the task t The detection sub-region corresponding to the y coordinate in the i-th batch.

[0022] A diffraction neural network system applied to the rotatable modulation layer optimization method based on the diffraction neural network, comprising: a continuous laser for providing a stable monochromatic coherent light source;

[0023] A focusing lens arranged at the outlet of the continuous laser;

[0024] A pinhole arranged at the focal position of the focusing lens and located behind the focusing lens;

[0025] A collimating lens arranged behind the pinhole, and an input layer arranged behind the collimating lens,

[0026] A multi-layer general solidification phase plate arranged behind the input layer for initial phase modulation of the input laser beam, and a rotating solidification phase plate arranged behind the multi-layer general solidification phase plate;

[0027] A detector arranged behind the rotating solidification phase plate;

[0028] ​Further comprising: an electrically controlled rotating table arranged between the input layer and the detector, used for rotating the rotating solidified phase plate.

[0029] The present application at least includes the following beneficial effects: by rotating the diffraction modulation layer, the optical calculation task under different complex scenes can be realized, the diffraction network does not need to be retrained, has the characteristics of multi-task reuse, only needs to adjust the diffraction modulation layer without re-optimizing the network parameters, and greatly reduces the computing resources and hardware cost.

[0030] Other advantages, objects, and features of the present application will be apparent from the following description, and will be understood by those skilled in the art. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 The composition schematic diagram of the diffraction neural network system in the present application;

[0032] The figure legend: 1, continuous laser, 2, focusing lens, 3, pinhole, 4, collimating lens, 5, input layer, 6, general solidified phase plate, 7, rotating solidified phase plate, 8, output layer. DETAILED DESCRIPTION

[0033] The present application will be further described in detail below with reference to the accompanying drawings, so that those skilled in the art can implement it according to the description. It should be understood that the terms such as "have", "contain" and "include" used herein do not exclude the presence or addition of one or more other elements or combinations thereof. It should be noted that in the description of the present application, the orientation or position relationship indicated by the terms is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0034] The present application is a rotatable modulation layer optimization method based on a diffraction neural network, comprising:

[0035] S1, loading a target image as an input layer of a diffraction optical network through a phase type spatial light modulator; the input layer of the diffraction optical network is loaded with a target image by the phase type spatial light modulator, the target image is a gray scale handwritten digit, the image is constrained to a phase range of 0 to by using a Sigmoid normalization function, and a target to be identified is loaded in turn using a phase type spatial light modulator;

[0036] S2, using a coherent light source to irradiate the input layer, so that the loaded phase target is transmitted into the diffractive optical network forwardly; in order to make the loaded phase target into the diffractive network, the input layer (input target) is illuminated by using a 532nm coherent light source, and under the action of the illumination light source, the target light field propagates forwardly, which propagates in accordance with the Rayleigh-Sommerfeld theorem, and the propagation form can be expressed as:

[0037]

[0038] In the above formula, represents the impulse response function, represents the complex light field of the first diffraction modulation layer at the spatial position r, represents the complex light field of the first +1 diffraction modulation layer at the spatial position r, represents the coordinate of the first diffraction modulation layer at the spatial position r, r represents the coordinate of the first +1 diffraction modulation layer at the spatial position r. r

[0039] S3, for different tasks, the phase of the multi-layer general diffractive modulation layer and the rotation diffractive modulation layer in the diffractive optical network is jointly optimized and trained, so as to obtain the phase parameters I and phase parameters II corresponding to the trained general diffractive modulation layer and rotation diffractive modulation layer; through the rotation diffractive modulation layer, multiple tasks are realized, and the physical rotation of the rotation layer is equivalent to rotating the phase pattern in the layer coordinate system by an angle θ in the opposite direction. The specific rotation parameter can be written as:

[0040]

[0041] In the above formula, represents the rotation matrix, (x, y) represents the layer coordinate system, and the complex transmission function of the first diffraction modulation layer can be expressed as:

[0042]

[0043] By using a phase basis function with C N group symmetry, when the phase satisfies , different sub-task channels are selected equivalently (which needs to be optimized with subsequent layers and output detection layout), and for the training sample of each task t, a rotation is introduced during forward propagation, and a translation / scaling disturbance can be superimposed to expand the effective FOV;​​

[0044] Step four: for each task, a specific rotation modulation layer is selected to perform phase joint optimization, for classification task, the goal of training is to maximize the energy of light intensity in a specific region on the detection plane, while reducing the energy outside the region. In the optimization process, the cross-entropy loss function is used to evaluate the difference between the label and the distribution of different sub-detector regions. Through multiple rounds of phase optimization, the recognition accuracy curve rises to no longer change, and the training is completed;

[0045] S4, after training, based on the phase parameters I and phase parameters II obtained in S3, a general curing phase plate of each general diffractive modulation layer and a rotating curing phase plate of a rotating diffractive modulation layer are printed to form a physically rotatable diffractive optical network for different tasks.

[0046] Wherein, the general diffractive modulation layer and the rotating diffractive modulation layer are used to load the trained modulation phase, each general diffractive modulation layer and rotating diffractive modulation layer consists of an array of 200*200 independent modulatable neurons, each neuron has a size of 8 microns, and the phase value is constrained between 0 and 2 To realize the optical interconnection between neurons in the neural network, the distance between the diffractive modulation layers is set to 4 cm, under this setting, all the optical signals emitted by all the neurons on the previous layer can be transmitted to all the neurons on the next diffractive modulation accurately, and this cascade mode can realize the optical interconnection between 400 million neurons. Then the complex transmission function of task t on the The complex transmission function of task t on the

[0047]

[0048] Wherein, θ is the mechanical rotation angle of the layer, , usually take 1 approximate no absorption, phase using bounded mapping to the rotation parameter training, then the trained is represented by the following formula:

[0049] ,

[0050] In the above formula, represents the th diffractive modulation layer at the corresponding mechanical rotation angle, represents the Sigmoid function, which is used to constrain the optimization parameter value to between 0 and 1.

[0051] Therefore, from the th diffractive modulation layer to the​ complex light field of a layer satisfies the following formula:

[0052]

[0053] In the above formula, represents a pulse response function, the complex light field of the first diffraction modulation layer at a spatial position r, represents the complex transmission function of the first diffraction modulation layer at a spatial position r when the mechanical rotation angle is the complex transmission function of the first diffraction modulation layer at a spatial position r when the mechanical rotation angle is

[0054] In the above scheme, the entire training uses supervised learning, and the error back propagation algorithm is used to optimize the phase parameters of the diffraction network modulation layer. The joint training and physical deployment for realizing “completing three classification tasks (MNIST handwritten digits, Fashion-MNIST fashion items, CIFAR-10) on the same set of diffractive optical neural networks (ONN) through rotatable modulation layers”.

[0055] Three types of data sets are simultaneously transmitted into the network as learning objects. Among them, the original size of each sample is 28×28, and the sampling bilinear interpolation expands the size of the original target from 28×28 to 150×150. In order to make the size of the input target and the diffraction modulation layer match, zero padding is performed around each input target. Then the intensity of the output plane complex field is represented by the following formula:

[0056]

[0057] In the above formula, represents the complex light field of the first diffraction modulation layer at a spatial position r;

[0058] Further, in order to identify the category of the target through the light intensity distribution on the detection plane, the detection plane is divided into 10 detection sub-regions, and the size of each detection sub-region is 10×10. After the network output light intensity, the length of the vector is calculated by summing the sub-regions on the 10 detection planes. Then the task t corresponds to the integral k of the first category (the first detection plane corresponds to a category). is represented by the following formula:

[0059] ,

[0060] In the above formula, ​​​is the set of all classes on the 10 probe planes, then after the Softmax function, the task t predicts the probability of each class is represented by the following equation:

[0061] .

[0062] After the forward transmission is completed, to further optimize the parameters of the network, the cross-entropy loss function is used to calculate the residual of the output vector and the One-hot encoding, and the loss value is updated by the back propagation algorithm to update the parameters of the diffractive modulation layer. Although the same set of diffractive modulation layers integrates two different recognition tasks, the two tasks cannot share the same phase parameters, because the rotation transformation of the diffractive modulation layer will arrange a new network structure to cope with the new calculation task. For the design of the loss function, the energy efficiency of the optical network needs to be considered here, although the cross-entropy loss function can optimize the classification performance to the best, but at the same time it will also cause the low energy utilization rate on the probe plane. Therefore, we use a hybrid loss function to alleviate the above problems. Specifically, the hybrid loss function is represented by the following equation:

[0063] L=L ce +L leak

[0064] In the above equation, L ce represents the main classification loss function, and

[0065]

[0066] In the above equation, is the weight, which is 1 by default, e is the One-hot encoding, represents the phase on the th diffractive modulation layer.

[0067] L leak is used to constrain the light intensity leakage of the non-target area, and

[0068]

[0069] In the above equation, B is the batch size, that is, the number of samples included in one training; is used to average the loss of all samples, I(r) is the intensity at the spatial position r, which is the distribution obtained on the probe plane after the light field passes through the diffractive neural network, β 1, β 2 are weight coefficients, used to balance the influence of the two constraint terms of "suppressing leakage energy" and "enhancing target energy";

[0070] Figure 1A diffraction neural network system is shown, comprising:

[0071] A continuous laser 1 for providing a stable monochromatic coherent light source, the continuous laser 1 generates a continuous wave of 532 nm, power 30 mW, monochromatic coherent light source;

[0072] A focusing lens 2 is arranged at the outlet of the continuous laser 1;

[0073] A pinhole 3 is arranged at the focal position of the focusing lens 2 and behind the focusing lens 2;

[0074] A collimating lens 4 is arranged behind the pinhole 3, and an input layer 5 is arranged behind the collimating lens 4,

[0075] A multi-layer general solidification phase plate 6 for initial phase modulation of the input laser beam is arranged behind the input layer 5, and a rotating solidification phase plate 7 is arranged behind the multi-layer general solidification phase plate 6;

[0076] A phase-type spatial light modulator (resolution 1920x1080, pixel 8 µm) is used to load the gray-scale image of a handwritten digit or a fashion item and phase-encode it as an output layer 8;

[0077] The multi-layer general diffraction modulation layer has a total of 5 layers, and each layer is a general solidification phase plate, and the distance between adjacent layers is 30 mm.

[0078] A rotating diffraction modulation layer is arranged behind the general diffraction modulation layer, and the distance between the rotating diffraction modulation layer and the general diffraction modulation layer is 30 mm. A rotating solidification phase plate is installed on a high-precision electrically controlled rotating table for rotating the steering of the last layer of rotating diffraction modulation layer to realize different computing tasks.

[0079] A CCD (2048x2048, pixel 6.5 µm) is arranged 100 mm behind the 5th layer for capturing the results of the diffraction network calculation.

[0080] A detector is arranged behind the rotating solidification phase plate;

[0081] Further comprising: an electrically controlled rotating table arranged between the input layer and the detector for rotating the rotating solidification phase plate.

[0082] Working principle:

[0083] In actual operation, the system first emits a stable laser beam from the continuous laser 1, which is focused through the focusing lens 2 and then spatially filtered through the pinhole 3. Then, the collimating lens 4 collimates the light beam into a parallel light beam and irradiates the input layer 5. The light beam passes through the input layer and enters the multi-layer general diffractive modulation layer and the rotating diffractive modulation layer formed by the multi-layer general solidified phase plate 6, the single-layer rotating solidified phase plate 7 and the electrically controlled rotating table for phase modulation. The modulated light beam propagates to the detector on the output layer 8, and the detector uses a CCD.

[0084] Among them, the multi-layer general diffractive modulation layer and the rotating diffractive modulation layer are responsible for phase modulation of the input laser beam. It can accurately adjust the wavefront according to the preset phase pattern to realize specific optical functions. In the system, the multi-layer general diffractive modulation layer is composed of 6 layers of diffractive modulation layers through 5 layers of general solidified phase plates and a single-layer rotating solidified phase plate of the rotating diffractive modulation layer. Each layer is a passive phase surface containing a programmable neuron structure, and its optimal phase distribution is obtained through deep learning training. The light field is transmitted in the form of free space propagation (Fresnel approximation) between layers, and the nonlinear mapping of phase to intensity is gradually completed in this process, finally forming a highly discriminative light intensity output mode, which can be used to accurately judge the input light field characteristics and complete complex optical tasks. When identifying different tasks, the last layer of the rotating solidified phase plate is rotated through the electrically controlled rotating table (not shown in the figure) to change the angle of the last layer of the rotating diffractive modulation layer, so as to realize the optical calculation task in different complex scenes without retraining the diffractive optical network, and has the characteristics of multi-task reuse.

[0085] Example 1: Handwritten digit recognition

[0086] Data preparation

[0087] 0-9 handwritten digit images are imported from the MNIST dataset, with 60,000 training images and 10,000 test images. Each image has an original size of 28x28, which is expanded to match the input SLM resolution (such as 256x256 or larger) through zero padding and interpolation, and is phase encoded.

[0088] Optical input: Load the phase information of the input digital image through the spatial light modulator, and modulate the incident 532 nm laser beam.

[0089] Diffractive propagation: the light field passes through the 6 layers of diffractive modulation layers in turn, the first 5 layers are fixed phase plates, and the 6th layer (SLM II) loads the reference phase , the rotation angle is set to ; on the output detection surface, for task A, the detection sub-regions corresponding to 10 non-overlapping discrimination windows are defined as , and each detection sub-region corresponds to a digital "0-9".

[0090] CCD images the output light field and calculates the intensity of each window , and the predicted class is the number corresponding to the window with the maximum intensity. In the inference stage, only need to fix SLM II at , and the number recognition can be realized.

[0091] Example 2: Fashion item dataset recognition

[0092] In the data preparation stage, 10 classes of clothing (such as shoes, bags, coats, etc.) are imported from the Fashion-MNIST dataset, with 60,000 training images and 10,000 test images. The images are also processed from 28x28 to the input SLM resolution and phase encoded.

[0093] The input sample image is loaded into the spatial light modulator, which modulates the laser wavefront. The light field passes through the 6th layer of diffraction modulation layer in turn, and the phase of the first 5 layers is the same as that of Example 1. The 6th layer SLM II keeps the phase template , but the rotation angle is set to On the output detection surface, for task B, the 10 discrimination windows are redefined as , and each detection sub-region corresponds to 10 classes of fashion items.

[0094] CCD images and integrates in each window area to get the energy distribution corresponding to each class. Similarly, the output class is determined by the window with the maximum intensity.

[0095] The training stage jointly optimizes the parameters (trained at the same time as Example 1), and in the inference stage, only need to rotate the 6th layer to , and the fashion item recognition can be completed.

[0096] In summary, in the whole system, the phase distribution of the first 5 layers is fixed, which serves both handwritten digit recognition and fashion item recognition. The switching of Example 2 only needs to rotate the last layer of the fixed phase plate from to . This method avoids retraining or replacing the phase plate and realizes the multi-task reuse of the same optical platform.

[0097] Although the embodiments of the present application have been disclosed as above, they are not limited to the application and implementation listed in the specification and examples, and can be fully applied to various fields suitable for the present application. Those skilled in the art can easily make other modifications, and therefore the present application is not limited to specific details and the figures shown and described herein, without departing from the general concept defined by the claims and their equivalent scope.​

Claims

1. A method for optimizing rotatable modulation layers based on diffraction neural networks, characterized in that, Comprise: S1, load a target image as an input layer of a diffractive optical network through a phase type spatial light modulator; S2, use a coherent light source to illuminate the input layer, so that the loaded phase target is transmitted forward into the diffractive optical network; S3, for different tasks, jointly optimize the phases of the multi-layer general diffractive modulation layer and the rotating diffractive modulation layer in the diffractive optical network to obtain the phase parameters I and phase parameters II corresponding to the trained general diffractive modulation layer and rotating diffractive modulation layer; S4, after training, print the general solidified phase plate of each general diffractive modulation layer and the rotating solidified phase plate of the rotating diffractive modulation layer based on the phase parameters I and phase parameters II obtained in S3, to construct a physically rotatable diffractive optical network for different tasks; In S3, the cross-entropy loss function is used to calculate the loss value of the output vector and the One-hot encoding, and the loss value is updated to the phase parameters of the diffractive modulation layer through the back propagation algorithm, and the cross-entropy loss function is characterized by the following formula: L = L ce + L leak In the above formula, L ce represents the main classification loss function, and ; In the above formula, is the weight coefficient of task t, e is One-hot encoding, represents the task t The phase on the i-th diffraction modulation layer, B represents the batch size, represents the probability of the mechanical rotation angle of task t in the i-th batch; represents the encoding of the y coordinate of task t in the i-th batch, CE() represents the loss function, i=1, 2…B;​ L leak represents the leakage loss, and ； In the above formula, β 1、 β 2 are weight coefficients, is a task t The light intensity at the spatial position r, r is the spatial position, denotes a task t The detection sub-region corresponding to the y coordinate in the i th batch.

2. The diffractive neural network-based rotatable modulation layer optimization method of claim 1, wherein, In S4, the general diffractive modulation layer and the rotating diffractive modulation layer are used to load the trained modulation phase, each of which consists of an array of 200*200 independently modulatable neurons, each neuron with a size of 8 microns, whose phase values are constrained between 0 and 2π by a Sigmoid function. between.

3. The diffractive neural network-based rotatable modulation layer optimization method of claim 1, wherein, In S1, the target image is a gray scale handwritten digit, which is constrained to the phase range of 0 to 2π using a Sigmoid normalization function. In S1, the target image is a gray scale handwritten digit, which is constrained to the phase range of 0 to 2π using a Sigmoid normalization function.

4. The diffractive neural network-based rotatable modulation layer optimization method of claim 1, wherein, In S2, a 532nm coherent light source is used to illuminate the input layer, so that the light field of the loaded phase target follows the Rayleigh-Sommerfeld theorem and is transmitted forward into the diffractive optical network, and the propagation form of the light field is characterized by the following formula: ； In the above formula, Represents the impulse response function. Indicates the first The complex optical field at spatial position r of a diffraction modulation layer Indicates the first The complex optical field at spatial position r of +1 diffraction modulation layer Indicates the first The diffraction modulation layer is located in space. r The coordinates of the location Indicates the first +1 diffraction modulation layer in spatial position r The coordinates of the location.

5. A diffractive neural network system applied in the method of optimizing a rotatable modulation layer based on a diffractive neural network according to any one of claims 1-4, characterized in that, Comprise: A continuous laser for providing a stable monochromatic coherent light source; A focusing lens arranged at the outlet of the continuous laser; A pinhole arranged at the focal point position of the focusing lens and located behind the focusing lens; A collimating lens arranged behind the pinhole, an input layer arranged behind the collimating lens, A multi-layer general solidified phase plate arranged behind the input layer for performing initial phase modulation on the input laser beam, and a rotating solidified phase plate arranged behind the multi-layer general solidified phase plate; A detector arranged behind the rotating solidified phase plate; Further comprising: an electrically controlled rotating table arranged between the input layer and the detector for rotating the rotating solidified phase plate.

Images