Design method and device of metasurface optical neural network

Abstract

The application discloses a design method and device of a metasurface optical neural network, and the design method comprises the following steps: obtaining an artificial neural network model and a metasurface optical neural network model, wherein the artificial neural network model is a trained model; selecting n intermediate layers in the artificial neural network model in the order from shallow to deep, one-to-one corresponding the n intermediate layers and n metasurface layers in sequence, and calculating an intermediate feature loss according to the similarity between the features output by each layer in the n intermediate layers and the features output by the corresponding metasurface layer; training the metasurface optical neural network model; and determining the process parameters of the metasurface according to the trained metasurface optical neural network model. The design method completely transplants the calculation capacity of the artificial neural network into the metasurface optical neural network, solves the problems of low design efficiency and low precision of the metasurface optical neural network, and is beneficial to the development of the metasurface optical neural network and the deployment in an environment with limited computing power and storage.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of deep learning and optoelectronic material design, and in particular to a design method and device of a metasurface optical neural network. BACKGROUND

[0002] The metasurface optical neural network can use light as an information carrier and use a two-dimensional metasurface material as a light controller. The metasurface optical neural network has characteristics such as high bandwidth, high interconnectivity, and parallel processing, and can make up for the high hardware requirements of an artificial neural network. In addition, the metasurface optical neural network also has a faster operation speed than an artificial neural network.

[0003] In the process of forming the present application, the inventors found that the design of the metasurface optical neural network has the following technical problems:

[0004] On the one hand, since the metasurface is a kind of subwavelength structure of artificial medium, the number of its regulation units can reach hundreds of millions, and the design of the metasurface depends on a large number of numerical simulation operations, and researchers need to manually optimize the numbers according to professional knowledge, which is time-consuming, so the design efficiency is low. On the other hand, due to the unreasonable design parameters of the metasurface and the limitation of the design accuracy of the metasurface, the existing metasurface optical neural network can only be applied to simple classification, such as digital classification, and cannot be used for classification in complex backgrounds, which is still insufficient compared with an artificial neural network.

[0005] Therefore, the existing metasurface optical neural network has low design efficiency and low accuracy, which restricts the development of the metasurface optical neural network. SUMMARY

[0006] To solve the above technical problems of the low design efficiency and low accuracy of the metasurface optical neural network, the purpose of the present application is to provide a design method and device of a metasurface optical neural network with higher design efficiency and better accuracy.

[0007] To achieve the above application purpose, an embodiment of the present application provides a design method of a metasurface optical neural network, the metasurface optical neural network comprising n layers of metasurfaces, and the design method comprising the following steps:

[0008] An artificial neural network model and a metasurface optical neural network model are obtained, wherein the artificial neural network model is a trained model, the artificial neural network model comprises a plurality of intermediate layers, the metasurface optical neural network model is a model converted from a metasurface optical neural network topology, the metasurface optical neural network model comprises n layers of metasurface layers, and the number of intermediate layers is not less than n layers;

[0009] An image set for training is obtained;

[0010] Obtain a loss function, wherein the loss function includes intermediate feature loss. In the artificial neural network model, n intermediate layers are selected in order from shallow to deep. The n intermediate layers are matched one-to-one with the n metasurface layers in sequence. The intermediate feature loss is calculated based on the similarity between the features output by each of the n intermediate layers and the features output by the corresponding metasurface layer.

[0011] The image set is input into the artificial neural network model and the metasurface optical neural network model respectively, and the metasurface optical neural network model is trained based on the loss function and the artificial neural network model.

[0012] The process parameters of the metasurface are determined based on the trained metasurface optical neural network model.

[0013] As a further improvement of the present invention, the artificial neural network model is a deep convolutional neural network model.

[0014] As a further improvement of the present invention, the artificial neural network model has n intermediate layers, and each intermediate layer corresponds to each metasurface layer in order from shallow to deep.

[0015] As a further improvement of the present invention, the number of intermediate layers of the artificial neural network model can be basically divided into n equal parts, wherein each part has m intermediate layers, and there are m-1 intermediate layers between the intermediate layers corresponding to two adjacent metasurface layers, wherein m is an integer not less than 2.

[0016] As a further improvement of the present invention, the method for calculating the intermediate feature loss includes:

[0017] Calculate the mean square error between the features output by each intermediate layer and the features output by the corresponding metasurface layer.

[0018] As a further improvement of the present invention, the loss function further includes a classification loss, and the method for calculating the classification loss includes:

[0019] The images in the image set are used by the metasurface optical neural network to generate classified images;

[0020] The classification loss is determined based on the difference between a specified classification region and other regions in the classification image.

[0021] As a further improvement of the present invention, the step of determining the classification loss based on the difference between the specified classification region and other regions in the classification image includes:

[0022] Based on the classification region specified in the classification image, a contrast region with the same size and shape as the classification region is determined in the background region outside the classification region;

[0023] Calculate the grayscale difference between the classification region and the comparison region at the same pixel position in the grayscale image corresponding to the classification image;

[0024] The classification loss is determined by statistically analyzing the grayscale differences corresponding to all pixels in the classification region, wherein the classification loss is the mean of all grayscale differences, the sum of all grayscale differences, the maximum value among all grayscale differences, or the minimum value among all grayscale differences.

[0025] As a further improvement of the present invention, the formula for calculating the loss function Loss is as follows:

[0026] Loss = W1*L1 + W2*L2 + ... + W n *L n +W n+1 *L c

[0027] Where W1, W2...Wn+1 are different custom weights, L1, L2...Ln are the intermediate feature losses corresponding to each metasurface layer, and Lc is the classification loss.

[0028] To achieve one of the above-mentioned objectives, an embodiment of the present invention provides a design apparatus for a metasurface optical neural network, wherein the metasurface optical neural network comprises n layers of metasurfaces, and the design apparatus includes:

[0029] The model acquisition module is used to acquire an artificial neural network model and a metasurface optical neural network model. The artificial neural network model is a trained model and includes several intermediate layers. The metasurface optical neural network model is a model converted from the topology of the metasurface optical neural network and includes n metasurface layers. The number of intermediate layers is not less than n.

[0030] The image acquisition module is used to acquire the image set for training.

[0031] The loss function acquisition module is used to acquire a loss function, wherein the loss function includes intermediate feature loss. In the artificial neural network model, n intermediate layers are selected in order from shallow to deep. The n intermediate layers are matched one-to-one with the n metasurface layers in sequence. The intermediate feature loss is calculated based on the similarity between the features output by each of the n intermediate layers and the features output by the corresponding metasurface layer.

[0032] The training module is used to input the image set into the artificial neural network model and the metasurface optical neural network model respectively, and train the metasurface optical neural network model based on the loss function and the artificial neural network model.

[0033] The process parameter generation module is used to determine the process parameters of the metasurface based on the trained metasurface optical neural network model.

[0034] To achieve one of the above-mentioned objectives, one embodiment of the present invention provides an electronic device, comprising:

[0035] Storage module, used to store computer programs;

[0036] The processing module, when executing the computer program, can implement the steps in the above-described design method of metasurface optical neural networks.

[0037] To achieve one of the above-mentioned objectives, an embodiment of the present invention provides a readable storage medium storing a computer program, characterized in that the computer program, when executed by a processing module, can implement the steps in the above-mentioned design method of metasurface optical neural networks.

[0038] Compared with the prior art, the present invention has the following beneficial effects: The design method and device of the metasurface optical neural network trains each layer of the metasurface optical neural network through a pre-trained artificial neural network, so that each layer of the metasurface optical neural network is fully trained, and finally the computing power of the artificial neural network is completely transferred to the metasurface optical neural network. The metasurface optical neural network has the ability to classify complex backgrounds comparable to that of the artificial neural network, thereby solving the problems of low design efficiency and low accuracy of the metasurface optical neural network, which is conducive to the development of the metasurface optical neural network and its deployment in environments with limited computing power and storage. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the structure of a metasurface optical neural network according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the structure of some basic elements of a metasurface according to an embodiment of the present invention;

[0041] Figure 3 This is a flowchart of a design method for a metasurface optical neural network according to an embodiment of the present invention;

[0042] Figure 4 This is a framework diagram of training a metasurface optical neural network model according to an embodiment of the present invention;

[0043] Figure 5 This is a framework diagram for training a metasurface optical neural network model according to another embodiment of the present invention;

[0044] Figure 6 This is a framework diagram for training a metasurface optical neural network model according to another embodiment of the present invention;

[0045] Figure 7 This is a schematic diagram of a module of a design device for a metasurface optical neural network according to an embodiment of the present invention;

[0046] Among them, 10 is the metasurface optical neural network model; 11 is the metasurface layer; 20 is the artificial neural network model; and 21 is the intermediate layer. Detailed Implementation

[0047] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0048] One embodiment of the present invention provides a design method and apparatus for metasurface optical neural networks that are more efficient and have better accuracy.

[0049] Metasurface Optical Neural Network

[0050] The metasurface optical neural network of this embodiment includes several layers of metasurfaces. These metasurface layers are programmable, forming a structure similar to a neural network, which greatly improves its expressive power. It can undertake at least part of the image feature extraction task. The imaging of the light signal after passing through the metasurface layers can directly obtain a classification image for classification. In addition, if the image output by the metasurface optical neural network is connected to other subsequent artificial neural networks, the classification preprocessing performed by the metasurface optical neural network before imaging can also speed up the computation of the subsequent artificial neural networks.

[0051] The structure of metasurfaces can be referenced Figure 1 As shown, the hypersurface can have one or more layers. The value of n in the diagram can be 1, 2, 3, 4, 5, or even more. Each layer of the hypersurface contains many primitives to be trained, such as... Figure 2 As shown, the geometric parameters of each primitive can be designed individually. These geometric parameters include the primitive's shape, period, and the corresponding dimensional and rotational angle parameters. Using the metasurface optical neural network design method and apparatus of this embodiment, the geometric parameters of each primitive can be obtained, and then process parameters can be generated based on these geometric parameters.

[0052] The primitives can be linear, cylindrical, cuboid, elliptical, hollowed-out elliptical bodies, hollowed-out cuboids, etc. Different shapes can correspond to some identical and some different dimensional and rotational parameters. Examples of identical parameters include periodicity and height, while differences include the radius of a cylinder, the radius and width of a cuboid, and the major and minor axes of an elliptical cylinder. Different primitives can have different rotational parameters. These primitive parameters collectively constitute the metasurface geometry.

[0053] Furthermore, metasurface optical neural networks can be used in conjunction with imaging systems. These systems include an aperture, lens group, and CMOS (Complementary Metal Oxide Semiconductor). The aperture collects light field information, the lens group forms the image, the metasurface controls light propagation, and the CMOS performs photoelectric conversion, recording the light intensity distribution. The aperture, lens group, several metasurface layers, and CMOS are arranged sequentially.

[0054] Design method of metasurface optical neural network

[0055] The following is combined with Figures 3 to 6 This invention provides a method for designing a metasurface optical neural network, which can generate geometric parameters, material parameters, etc. of primitives on several layers of metasurface in the metasurface optical neural network.

[0056] Although this application provides method operation steps as shown in the following embodiments or flowcharts, the execution order of steps in which there is no necessary causal relationship logically, based on conventional or non-inventive effort, is not limited to the execution order provided in the embodiments of this application. For example, the acquisition order of steps S10, S20, and S30 below can be arbitrarily adjusted, or performed simultaneously, without distinguishing the chronological order.

[0057] Specifically, the design method of the metasurface optical neural network in this embodiment is as follows: Figure 3 As shown, it includes the following steps:

[0058] Step S10: Obtain the artificial neural network model and the metasurface optical neural network model.

[0059] Artificial neural network models can be pre-trained models with high accuracy, capable of classifying images with complex backgrounds and accurately classifying a specified classification task. During training, an artificial neural network model can first be trained using real images relevant to the specific classification task to obtain an initial model. Then, in the pre-training phase, a GAN network is used to adversarially train the initial model to obtain an artificial neural network model with accurate classification capabilities for that task.

[0060] In this embodiment, the artificial neural network model is a deep convolutional neural network model.

[0061] The metasurface optical neural network model is a model derived from the topological transformation of the aforementioned metasurface optical neural network. This model comprises n metasurface layers, each corresponding to a metasurface in the network. Training the model allows for the design of the metasurface layers, thereby deriving the specific structure of the metasurface. Additionally, other non-geometric parameters, such as material parameters, can also be included as training parameters.

[0062] Different complex background tasks can correspond to different metasurface optical neural networks. In other words, different classification tasks can have different numbers of layers, materials, geometric parameters of primitives, etc.

[0063] The framework diagram of the training process of the metasurface optical neural network model 10 is as follows: Figures 4 to 6 As shown, the artificial neural network model 20 includes several intermediate layers 21. The number of intermediate layers 21 in the metasurface optical neural network model 10 is not less than n layers. The number of intermediate layers 21 can be equal to n layers or greater than n layers, depending on actual needs. Preferably, in this embodiment, the number of intermediate layers 21 is greater than n, or even much greater than n. This way, by training the complex artificial neural network model 20 and then transferring its classification ability to the metasurface optical neural network model 10, the classification ability of the metasurface optical neural network can be improved. Figure 4 As shown, the multilayer intermediate layer 21 corresponds to the metasurface layer 11 of one layer.

[0064] In addition, in other embodiments, such as Figure 5 As shown, the number of intermediate layers 21 in the artificial neural network model 20 is also n. Each intermediate layer 21 corresponds to each metasurface layer 11 in order from shallow to deep. The number of intermediate layers 21 is equal to the number of metasurface layers 11.

[0065] And, in other embodiments, such as Figure 6 As shown, the number of intermediate layers 21 in the artificial neural network model 20 can be roughly divided into n equal parts, with m intermediate layers 21 in each part. There are also m-1 intermediate layers 21 between adjacent intermediate layers 21 corresponding to the metasurface layers 11, where m is an integer not less than 2. Figure 6 In this case, every two intermediate layers 21 correspond to one metasurface layer 11. Alternatively, every three intermediate layers 21 can correspond to one metasurface layer 11, and so on, including more cases where m is greater than 3.

[0066] Step S20: Obtain the image set for training.

[0067] The image set includes a large number of images used to train the model. Obtaining the image set may also include steps S21 to S23:

[0068] Step S21: Obtain the sample image set;

[0069] Step S22: Expand the sample image set to generate an image set, wherein the number of images in the image set meets the requirements of gradient backpropagation optimization of the metasurface optical neural network model.

[0070] Step S23: Normalize the image set to obtain a standardized image set that can be used for model training.

[0071] The sample image set obtained in step S21 can contain dozens or hundreds of samples. These samples are images that need to be classified against complex backgrounds. Then, these images are preprocessed, such as color space conversion, image sharpening, and image blurring.

[0072] Then, step S22 expands the number of these images. For example, image processing techniques such as interpolation, flipping, and stretching are used to augment the collected samples, increasing the number of training images to tens of thousands, such as hundreds of thousands. This method of expanding the number of images has been experimentally verified and meets the requirements for model training.

[0073] Then, in step S23, the expanded image is normalized to obtain a standard image of the same form.

[0074] In addition, the image set can include training and validation datasets to support different training tasks.

[0075] Step S30: Obtain the loss function.

[0076] The loss function can calculate the loss value, which can then guide the learning and adjustment of the model. Through the gradient backpropagation method, the gradient calculated by the loss function is passed to the various parameters of the model, thereby updating the parameters and making the classification ability of the metasurface optical neural network model gradually approach that of the artificial neural network model, thus improving the performance of the metasurface optical neural network model.

[0077] The loss function in this embodiment includes intermediate feature loss, and the method for calculating intermediate feature loss includes:

[0078] Step S31: Select n intermediate layers in the artificial neural network model in the order from shallow to deep, correspond the n intermediate layers to the n metasurface layers one by one in sequence, and calculate the intermediate feature loss according to the similarity degree between the features output by each of the n intermediate layers and the features output by the corresponding metasurface layer.

[0079] As shown in Figures 4 to 6 , for the feature T output by the intermediate layer 21 corresponding to the metasurface layer 11 and the feature S output by this metasurface layer 11, calculate the similarity degree between T and S. In this embodiment, the mean square error between the feature T output by each intermediate layer 21 and the feature S output by the corresponding metasurface layer 11 can be calculated. If n is greater than 1, the mean square errors corresponding to multiple metasurface layers 11 are added. If the loss function only includes the intermediate feature loss, the calculation formula of the loss function Loss is:

[0080] Loss = W1 * L1 + W2 * L2 + …… + W n *L n , (Formula 1)

[0081] where, W1, W2 …… Wn are different user-defined weights, and L1, L2 …… Ln are the intermediate feature losses corresponding to each metasurface layer 11.

[0082] Furthermore, the loss function further includes a classification loss. The regional intensity of the classification image generated by the metasurface optical neural network can be compared, and then the classification loss can be designed. The classification loss can be used to calculate the gap between the classification region and the background region in the classification image. For example Figures 4 to 6 as shown, the area A1 in the classification image is used as the classification region. For example, when performing face recognition, the classification result of the target user is 1, and the classification result of the non-target user is 0, then the values in the area A1 are as close to 1 as possible; the areas outside the area A1 are used as the background regions, and the values in these regions are as close to 0 as possible. The classification loss can compare the differences between the area A1 and the areas outside the area A1. Ideally, the closer the difference value is to 1, the higher the classification accuracy.

[0083] The calculation method of the classification loss includes:

[0084] Step S321: Based on the specified classification region in the classification image, determine a comparison region with the same size and shape as the classification region in the background region outside the classification region;

[0085] Step S322: Calculate the gray level difference of the same pixel position of the classification region and the comparison region on the gray scale image corresponding to the classification image;

[0086] Step S323: Calculate the grayscale difference values ​​corresponding to all pixels in the classification region to determine the loss function.

[0087] In step S321, corresponding Figures 4 to 6 A1 is selected as the classification region, and A2, which has the same size and shape as A1, is selected as the comparison region. At this time, there is a one-to-one correspondence between the pixels in regions A1 and A2. Step S322 calculates the difference in grayscale value between each pixel in region A1 and the pixel in region A2. Step S323 designs a loss function that is statistically significant in comparing the differences between A1 and A2.

[0088] Furthermore, the classification loss includes parameters such as the mean of all grayscale differences, the sum of all grayscale differences, the maximum value among all grayscale differences, or the minimum value among all grayscale differences.

[0089] In this embodiment, the formula for calculating the classification loss Lc can be:

[0090] L c =W 11 *Mean+W 12 *Sum+W 13 *Max+W 14 *Min, (Formula 2)

[0091] Here, ΔA is defined as 1 - |A1 - A2|, where |A1 - A2| is the difference in grayscale value at the same pixel position in regions A1 and A2, the set of all pixels ΔA is {ΔA}, Mean is the mean of all elements in {ΔA}, Sum is the sum of all elements in {ΔA}, Max is the maximum value in {ΔA}, Min is the minimum value in {ΔA}, and W11, W12, W13, and W14 are the respective weight coefficients.

[0092] The specific values ​​of W11, W12, W13, and W14 can be designed by combining the process manufacturer and the product's working environment. Adjusting the appropriate weights can increase the robustness of the model.

[0093] Furthermore, Formula 2 is only one example of the classification loss Lc. The classification loss Lc can also include only a portion of the values ​​in this formula. For example, the classification loss Lc can also be L c =W 11 *Mean+W 14 *Min, or L c =W 12 *Sum+W 13 *Max, or L c =W 11 *Mean+W 13 *Max+W14 *Min, etc., and Lc can be any one of Mean, Sum, Max, and Min.

[0094] Furthermore, the classification loss Lc can also pursue the increase of A1 and the decrease of A2, or it can be minimized simultaneously with the gray values ​​in the A2 region by using a mathematical function, such as cos(π / 2*A1) for the gray values ​​in the A1 region.

[0095] Furthermore, the loss function combines both the intermediate feature loss and the classification loss mentioned above, that is, it combines both Formula 1 and Formula 2 mentioned above. The formula for calculating the loss function Loss is:

[0096] Loss = W1*L1 + W2*L2 + ... + W n *L n +W n+1 *L c

[0097] Where W1, W2...Wn, Wn+1 are different custom weights, L1, L2...Ln are the intermediate feature losses corresponding to each metasurface layer, and Lc is the classification loss.

[0098] Furthermore, steps S41 to S44 may be included before training the model:

[0099] Step S41: Initialize each hyperparameter. For example, the light source, metasurfaces, spatial configuration between metasurfaces, and physical parameters of the simulated CMOS.

[0100] Step S42: Initialize the trainable parameters. For example, initialize the geometric parameters of the metasurface layer.

[0101] Step S43: Optimizer selection. The optimizer can be the Adam optimizer, which takes the model's parameters and learning rate as input.

[0102] Step S44: Setting parameters for dynamic adjustment of the learning rate. Reducing the model's learning rate after each training period can prevent overfitting after multiple training iterations. In this embodiment, a stepped learning rate scheduler can be used to adjust the learning rate, reducing the learning rate after a certain number of training iterations using a stepped learning rate scheduler.

[0103] Step S50: Input the image set into the artificial neural network model and the metasurface optical neural network model respectively, and train the metasurface optical neural network model based on the loss function and the artificial neural network model.

[0104] Step S60: Determine the process parameters of the metasurface based on the trained metasurface optical neural network model.

[0105] This embodiment transfers the classification ability of each layer of the artificial neural network model to the metasurface of the metasurface optical neural network, so that a metasurface with excellent classification ability in complex backgrounds can be obtained without the need for specific design of the metasurface structure.

[0106] After the above training is completed, if the trainable parameters are the geometric parameters of the metasurface's primitives, then the corresponding geometric parameters of the primitives can be obtained. If the trainable parameters are more hardware parameters of the metasurface optical neural network, then more hardware parameters can be obtained. Taking the obtained geometric parameters of the primitives as an example, a photomask data format file (GDS, Graphical Data System) is then generated. This is a file format used to describe and record the metasurface layout, including information such as the physical layout and feature parameters of the metasurface. This serves as the basis and guide for subsequent manufacturing processes. Then, the corresponding hardware structure of the metasurface optical neural network can be produced, and the process department can create a metasurface lens that matches the parameters of the metasurface designed by the design method of the metasurface optical neural network.

[0107] Compared with the prior art, this embodiment has the following beneficial effects:

[0108] This design method for metasurface optical neural networks trains each layer of the metasurface optical neural network using a pre-trained artificial neural network, ensuring that each layer is fully trained. Ultimately, the computational power of the artificial neural network is completely transferred to the metasurface optical neural network, giving it a classification capability for complex backgrounds comparable to that of artificial neural networks. This solves the problems of low design efficiency and low accuracy in metasurface optical neural networks, which is beneficial for their development and deployment in environments with limited computing power and storage.

[0109] Design device for metasurface optical neural networks

[0110] In one embodiment, a design apparatus for a metasurface optical neural network is provided, such as... Figure 7 As shown. The design device for this metasurface optical neural network includes the following modules, and the specific functions of each module are as follows:

[0111] The model acquisition module is used to acquire an artificial neural network model and a metasurface optical neural network model. The artificial neural network model is a trained model and includes several intermediate layers. The metasurface optical neural network model is a model converted from the topology of the metasurface optical neural network and includes n metasurface layers. The number of intermediate layers is not less than n.

[0112] The image acquisition module is used to acquire the image set for training.

[0113] The loss function acquisition module is used to acquire a loss function, wherein the loss function includes intermediate feature loss. In the artificial neural network model, n intermediate layers are selected in order from shallow to deep. The n intermediate layers are matched one-to-one with the n metasurface layers in sequence. The intermediate feature loss is calculated based on the similarity between the features output by each of the n intermediate layers and the features output by the corresponding metasurface layer.

[0114] The training module is used to input the image set into the artificial neural network model and the metasurface optical neural network model respectively, and train the metasurface optical neural network model based on the loss function and the artificial neural network model.

[0115] The process parameter generation module is used to determine the process parameters of the metasurface based on the trained metasurface optical neural network model.

[0116] It should be noted that for details not disclosed in the design apparatus of the metasurface optical neural network in the embodiments of the present invention, please refer to the details disclosed in the design method of the metasurface optical neural network in the embodiments of the present invention.

[0117] The design apparatus for metasurface optical neural networks may further include computing devices such as computers, laptops, handheld computers, and cloud servers, as well as, but not limited to, processing modules, storage modules, and computer programs stored in the storage modules and executable on the processing modules, such as the aforementioned metasurface optical neural network design method program. When the processing module executes the computer program, it implements the steps in the various metasurface optical neural network design method embodiments described above, for example... Figure 3 The steps are shown.

[0118] In addition, the present invention also proposes an electronic device, which includes a storage module and a processing module. When the processing module executes the computer program, it can implement the steps in the above-mentioned metasurface optical neural network design method, that is, implement the steps in any one of the technical solutions in the above-mentioned metasurface optical neural network design method.

[0119] The electronic device can be part of a design device integrated into a metasurface optical neural network, a local terminal device, or part of a cloud server.

[0120] The processing module can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processing module is the control center of the metasurface optical neural network design device, connecting all parts of the design device through various interfaces and lines.

[0121] The storage module can be used to store the computer programs and / or modules. The processing module realizes various functions of the metasurface optical neural network design device by running or executing the computer programs and / or modules stored in the storage module and calling the data stored in the storage module. The storage module may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system, at least one application program required for a function, etc. In addition, the storage module may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0122] For example, the computer program can be divided into one or more modules / units, which are stored in a storage module and executed by a processing module to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in a metasurface optical neural network design device.

[0123] Furthermore, one embodiment of the present invention provides a readable storage medium storing a computer program that, when executed by a processing module, can implement the steps in the above-described metasurface optical neural network design method, that is, implement the steps in any one of the technical solutions in the above-described metasurface optical neural network design method.

[0124] If the modules integrated in the metasurface optical neural network design method are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processing module, it can implement the steps of the above-described method embodiments.

[0125] The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording media, U disks, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0126] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0127] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A design method for a metasurface optical neural network, wherein the metasurface optical neural network comprises n layers of metasurfaces, characterized in that, The design method includes the following steps: Obtain an artificial neural network model and a metasurface optical neural network model, wherein the artificial neural network model is a trained model and includes several intermediate layers; the metasurface optical neural network model is a model topologically transformed from the metasurface optical neural network and includes n metasurface layers, each metasurface layer corresponding to the metasurface of the metasurface optical neural network; the trainable parameters of the metasurface layers include the geometric parameters of the primitives of the metasurface; and the number of intermediate layers is not less than n. Obtain the image set for training; Obtain a loss function, wherein the loss function includes intermediate feature loss. In the artificial neural network model, n intermediate layers are selected in order from shallow to deep. The n intermediate layers are matched one-to-one with the n metasurface layers in sequence. The intermediate feature loss is calculated based on the similarity between the features output by each of the n intermediate layers and the features output by the corresponding metasurface layer. The image set is input into the artificial neural network model and the metasurface optical neural network model respectively. Based on the loss function and the artificial neural network model, the metasurface optical neural network model is trained to transfer the classification ability of each layer of the artificial neural network model to the metasurface of the metasurface optical neural network. The process parameters of the metasurface are determined based on the trained metasurface optical neural network model.

2. The design method of the metasurface optical neural network according to claim 1, characterized in that, The artificial neural network model is a deep convolutional neural network model.

3. The design method of the metasurface optical neural network according to claim 1, characterized in that, The artificial neural network model has n intermediate layers, and each intermediate layer corresponds to each metasurface layer in order from shallow to deep.

4. The design method of the metasurface optical neural network according to claim 1, characterized in that, The number of intermediate layers in the artificial neural network model is divided into n equal parts, with m intermediate layers in each part. There are m-1 intermediate layers between the intermediate layers corresponding to two adjacent metasurface layers, where m is an integer not less than 2.

5. The design method of the metasurface optical neural network according to claim 1, characterized in that, The method for calculating the intermediate feature loss includes: Calculate the mean square error between the features output by each intermediate layer and the features output by the corresponding metasurface layer.

6. The design method of the metasurface optical neural network according to claim 1, characterized in that, The loss function further includes a classification loss, and the method for calculating the classification loss includes: The images in the image set are used by the metasurface optical neural network to generate classified images; The classification loss is determined based on the difference between a specified classification region and other regions in the classification image, wherein the other regions are contrast regions of the same size and shape as the classification region, determined in the background region outside the classification region.

7. The design method for metasurface optical neural networks according to claim 6, characterized in that, Determining the classification loss based on the difference between a specified classification region and other regions in the classification image includes: Calculate the grayscale difference between the classification region and the comparison region at the same pixel position in the grayscale image corresponding to the classification image; The classification loss is determined by statistically analyzing the grayscale differences corresponding to all pixels in the classification region, wherein the classification loss is the mean of all grayscale differences, the sum of all grayscale differences, the maximum value among all grayscale differences, or the minimum value among all grayscale differences.

8. The design method of the metasurface optical neural network according to claim 6, characterized in that, The formula for calculating the loss function Loss is as follows: Where W1, W2...Wn+1 are different custom weights, L1, L2...Ln are the intermediate feature losses corresponding to each metasurface layer, and Lc is the classification loss.

9. A design apparatus for a metasurface optical neural network, the metasurface optical neural network comprising n layers of metasurfaces, characterized in that, The design device includes: The model acquisition module is used to acquire an artificial neural network model and a metasurface optical neural network model. The artificial neural network model is a trained model and includes several intermediate layers. The metasurface optical neural network model is a model topologically transformed from the metasurface optical neural network and includes n metasurface layers. Each metasurface layer corresponds to the metasurface of the metasurface optical neural network. The trainable parameters of each metasurface layer include the geometric parameters of the primitives of the metasurface. The number of intermediate layers is not less than n. The image acquisition module is used to acquire the image set for training. The loss function acquisition module is used to acquire a loss function, wherein the loss function includes intermediate feature loss. In the artificial neural network model, n intermediate layers are selected in order from shallow to deep. The n intermediate layers are matched one-to-one with the n metasurface layers in sequence. The intermediate feature loss is calculated based on the similarity between the features output by each of the n intermediate layers and the features output by the corresponding metasurface layer. The training module is used to input the image set into the artificial neural network model and the metasurface optical neural network model respectively, and train the metasurface optical neural network model based on the loss function and the artificial neural network model, so as to transfer the classification ability of each layer of the artificial neural network model to the metasurface of the metasurface optical neural network. The process parameter generation module is used to determine the process parameters of the metasurface based on the trained metasurface optical neural network model.

10. An electronic device, characterized in that, include: Storage module, used to store computer programs; The processing module, when executing the computer program, can implement the steps in the design method of the metasurface optical neural network according to any one of claims 1 to 8.

11. A readable storage medium storing a computer program, characterized in that, When executed by the processing module, the computer program can implement the steps in the design method of the metasurface optical neural network according to any one of claims 1 to 8.

Images