Multi-dimensional multiplexing holographic metasurface rapid design method based on deep transfer learning

Abstract

The invention discloses a multi-dimensional multiplexing holographic metasurface rapid design method based on deep transfer learning. According to the method, a self-supervised training-testing architecture is realized by using deep learning, so that the dynamic learning ability of artificial intelligence is exerted, a time-consuming iteration process is not needed after training, the metasurface structure parameters are directly reconstructed according to the determined network weight, and the design efficiency of the metasurface for the multi-dimensional multiplexing hologram is greatly improved. The design efficiency is greatly improved; residual convolution and cavity pyramid pooling operation are introduced in the metasurface structure parameter reconstruction process to enhance the coding performance, and the design effect of a metasurface hologram is improved; by introducing a multi-wavelength dual-polarization deep neural sub-network, the defect that an existing deep learning metasurface design method is only limited to monochromatic images is overcome; line width disturbance of the metasurface structure is introduced into the network in advance to simulate errors, the influence caused by the process precision problem can be relieved, and higher robustness is shown.

Description

Technical Field

[0001] The present invention relates to the field of metasurface holographic display, and in particular to a method for rapid design of multi-dimensional multiplexed holographic metasurfaces based on deep transfer learning. Background Art

[0002] Metasurface is a tiny planar device composed of artificial microstructures. Its huge design flexibility enables it to accurately manipulate multiple dimensions of light waves, such as amplitude, phase, polarization, and diffraction distance, in the subwavelength range. Compared with traditional optical devices, metasurfaces have multiple advantages such as high compactness and strong controllability. The forward design of metasurfaces refers to calculating the phase plate according to the target, and then searching for the best unit at each position in the structure library obtained by pre-simulation, so as to establish the structural parameters of the whole piece. This method is simple in concept, but it is difficult to consider the manipulation of multi-dimensional optical parameters at the same time, and spatial resolution or temporal resolution must be sacrificed to reuse dimensions. In contrast, the end-to-end inverse design method iteratively optimizes the unit structure of the metasurface by comparing the difference between the output and the target, showing better multi-dimensional design performance. However, both forward design and inverse design methods are based on repetitive iterative processes, and each design requires a large number of cyclic numerical calculations as support. When faced with the design task of multi-dimensional reused metasurfaces, even a small change in the target requires tens of minutes of recalculation, and even falls into convergence dilemmas such as local optimal solutions. The current low design efficiency limits the engineering application potential of metasurfaces.

[0003] In recent years, the development of artificial intelligence has provided new potential for metasurface design. Computers simulate the hierarchical learning process of human brain neurons through deep learning. After training, there is no need for iteration and the results are given in real time according to the input. Researchers at home and abroad have proposed two solutions: introducing deep neural networks (DNNs) in reverse design to solve the differentiability problem of the super-atom simulation library, and directly using deep learning to build an end-to-end framework for rapid design of metasurface holography. The former shows excellent multi-dimensional design effects, but it still belongs to the category of reverse design and requires a lot of time for iterative optimization; the latter combines the forward prediction network and the reverse design network to realize the training-testing artificial intelligence computing paradigm, which effectively reduces the design time. However, the design dimension of the existing technology is limited to monochrome (single wavelength) images, and the design effect is poor, with obvious crosstalk and noise distortion between the channels. Therefore, how to improve the efficiency of metasurface design while ensuring the quality of multi-dimensional reuse is an unresolved problem in the field of metasurface holography. Summary of the invention

[0004] In response to the problems existing in the above-mentioned prior arts, the present invention proposes a rapid design method for multi-dimensional multiplexed holographic metasurfaces based on deep transfer learning, which solves the problems of low time efficiency of metasurface design, poor design effect, and limited multiplexing dimension. Multi-dimensional multiplexed holography is realized by using metasurfaces without sacrificing spatial resolution.

[0005] The multi-dimensional multiplexed holographic metasurface rapid design method based on deep transfer learning of the present invention comprises the following steps:

[0006] 1) Designing super atoms:

[0007] The metasurface includes a substrate and a plurality of nanopillars. The surface of the substrate is discretized into L×W identical two-dimensionally arranged units. A nanopillar is arranged at the center of each unit, that is, there are L×W nanopillars in total. A unit with a nanopillar is called a metaatom. The cross-sectional size of the nanopillar is variable, and the height of all nanopillars is fixed. The metaatom modulates the incident light field, and changes the outgoing light field by changing the structural parameters of the metaatom, that is, the cross-sectional size of the nanopillar.

[0008] 2) Construct a training dataset for the multi-wavelength dual-polarization deep neural sub-network:

[0009] The incident light field is incident from the bottom of the substrate to the metasurface, and the meta-atom modulates the incident light field; the finite element analysis is used to calculate the output light field after the incident light field passes through a meta-atom; the structural parameters of the meta-atom are traversed, and multiple different wavelengths and two polarization states of the incident light field are traversed. The two polarization states are perpendicular to each other, and the corresponding output light fields under multiple different wavelengths and two polarization states of the incident light field are obtained after being modulated by meta-atoms with different structural parameters, which are used as the training data set of the multi-wavelength dual-polarization deep neural sub-network;

[0010] 3) Training multi-wavelength dual-polarization deep neural sub-network:

[0011] Establish a multi-wavelength dual-polarization deep neural sub-network, take the known structural parameters of the super-atom as input, take the corresponding outgoing light field after super-atom modulation under multiple different wavelengths and two polarization states of the incident light field as output, use the training data set obtained in step 2) to train the multi-wavelength dual-polarization deep neural sub-network, and freeze the weights of the trained multi-wavelength dual-polarization deep neural sub-network, that is, in subsequent applications, the weights of the multi-wavelength dual-polarization deep neural sub-network remain unchanged;

[0012] 4) Building a deep learning hypersurface design network:

[0013] The multi-dimensional multiplexed target hologram is used as the input of the deep learning metasurface design network, and the structural parameters of each meta-atom of the metasurface are used as the output; the multi-dimensional multiplexed target hologram includes N single holograms, the number of pixels of each single hologram is L×W, which is equal to the number of discretized units of the metasurface, and the number of pixels of the multi-dimensional multiplexed target hologram is L×W×N. A set of wavelengths, polarization states, and diffraction distances corresponds to a single hologram, and each single hologram corresponds to a coding channel. N is the number of coding channels of the multi-dimensional multiplexed target hologram, which is the sum of the number of coding channels of the output light field in the three dimensions of wavelength, polarization state, and diffraction distance.

[0014] The deep learning metasurface design network includes an embedding layer, an encoder-decoder layer, a structure output layer, a multi-wavelength dual-polarization deep neural sub-network, and an angular spectrum diffraction algorithm:

[0015] The embedding layer encodes the multi-dimensional multiplexed target hologram and converts the multi-dimensional multiplexed target hologram into a computer-recognizable L×W×C feature tensor, which is transmitted to the encoder-decoder layer; where C is the number of output channels of the embedding layer;

[0016] The encoder-decoder layer extracts, maps and reconstructs the feature tensor, obtains the reconstructed features and passes them to the structure output layer;

[0017] Through the structure output layer, all the reconstructed features are fitted and the structure parameter tensor of L×W×K is output, where K is the number of parameters of the cross-sectional dimensions; the structure parameter tensor represents the structural parameters of each superatom of the supersurface;

[0018] The multi-wavelength dual-polarization deep neural sub-network calculates the modulation response of each super-atom on the metasurface to the incident light field according to the structural parameters of each super-atom, obtains the outgoing light field at different wavelengths and polarization states, and outputs it in the form of complex amplitude, i.e., real part and imaginary part;

[0019] The angular spectrum diffraction algorithm is used to obtain the diffraction image of the outgoing light field at different diffraction distances, that is, the current hologram corresponding to the currently designed metasurface is obtained; the pixel sampling rate of angular spectrum diffraction is the same for light of different wavelengths, which ensures the realization of color holographic images;

[0020] 5) Self-supervised training of deep learning hypersurface design network:

[0021] Construct a multi-channel composite loss function, based on the input multi-dimensional multiplexed target hologram and the current hologram calculated in step 4), perform loss calculation according to the multi-channel composite loss function, and self-supervise the training of the deep learning metasurface design network; the deep learning metasurface design network has the characteristics of self-supervision, and does not need to manually construct labels. It can be trained by only inputting the multi-dimensional multiplexed target hologram to obtain a trained deep learning metasurface design network. The deep learning metasurface design network outputs a L×W×K structural parameter tensor representing the structural parameters of each metaatom of the metasurface, and the metasurface design is completed;

[0022] 6) Applying Deep Learning Hypersurface Design Network Using Transfer Learning:

[0023] After obtaining the trained deep learning metasurface design network, the multi-dimensional multiplexed target hologram is used as input to obtain the structural parameters of each meta-atom on the metasurface, that is, to design the metasurface; after the deep learning metasurface design network training is completed, the number of encoding channels is fixed. When faced with the metasurface design task of multi-dimensional multiplexed target holograms with different numbers of encoding channels, rapid switching is performed through transfer learning methods.

[0024] For the same number of target hologram dimensions and the same number of coding channels, the same deep learning metasurface design network is used; for the same number of target hologram dimensions but different numbers of coding channels and different numbers of target hologram dimensions (different dimensions necessarily mean different numbers of coding channels), different deep learning metasurface design networks are used, but the architecture of the deep learning metasurface design network is the same, and only the specific number of coding channels is changed. Further, the present invention also includes a transfer learning method, for multi-dimensional multiplexed target holograms with different numbers of coding channels, a new deep learning metasurface design network is constructed using a transfer learning method, the same architecture is used in the new deep learning metasurface design network, the weights corresponding to each node of the encoder-decoder layer of the original deep learning metasurface design network are extracted, and they are transferred to the new deep learning metasurface design network, and the weights are frozen, and fine-tuning training is performed in the new deep learning metasurface design network to make the network converge quickly.

[0025] The network in the present invention adopts transfer learning when switching different application scenarios (the reuse dimension and the number of target channels change), so that it can effectively inherit the ability to understand image features that has been acquired through training in previous tasks. The transfer learning method enables the network to only perform fine-tuning training when facing new application tasks, without having to re-perform time-consuming full training, effectively improving the training efficiency of the network. Specifically, transfer learning uses the encoder-decoder layer as the migration subject: first, the network is trained with a universal single-dimensional three-channel color (red, green, and blue) hologram. The single dimension refers to one dimension of wavelength, and the three-channel color refers to red, green, and blue. The task uses natural scene images with complex textures and rich features as the training set. After the training is completed, the weights of each node in the structure are determined; for the design tasks of super surfaces in different application scenarios, such as multi-dimensional multiplexed holograms, the determined weights of each node are migrated to a new network with the same architecture, and their weights are frozen. Before and after the encoder-decoder layer, the tensor size of the encoding layer and the output layer is flexibly adjusted. When the determined three-channel weights are migrated to a network with different encoding channels, the dimension matching can still be guaranteed. Fine-tuning training is then carried out to achieve rapid convergence. After training, the network reaches a design time of sub-second (<1s), and the efficiency is increased by more than 1,000 times with the same design effect.

[0026] In step 1), the substrate is made of transparent material, fused quartz (SiO 2 ), sapphire (Al2O3) or ordinary glass (SiO2); nanorods are made of dielectric materials, titanium dioxide nanorods (TiO 2 ), silicon nitride (Si3N4), silicon (Si) or gallium nitride (GaN). The unit period P is 300nm~500nm, the cross-sectional dimension should be less than 50nm of the unit period, and the height is fixed at 500~800nm. One superatom corresponds to one pixel of the target hologram. The cross-sectional shape of the nanocolumn is an axisymmetric figure, such as a rectangle, circle or ellipse. The cross-sectional dimensions of a rectangular nanocolumn are the length and width; the cross-sectional dimension of a circular nanocolumn is the radius; the cross-sectional dimensions of an elliptical nanocolumn are the major axis and the minor axis. L is a natural number ≥100, and W is a natural number ≥100.

[0027] In step 2), the step size of the cross-sectional size is 1 to 10 nm. If the interval is too large, for example 20 nm, the data obtained will be correspondingly reduced, and the calculation accuracy of the outgoing light field by the trained multi-wavelength dual-polarization deep neural sub-network will decrease. The wavelength of the incident light field is 450 to 800 nm, covering the visible light band. 480 nm blue light, 532 nm green light, 633 nm red light and 680 nm deep red light are actually used. The outgoing light field includes the real and imaginary parts of the complex amplitude of the light field. The outgoing light field has the same wavelength and polarization state as the incident light field.

[0028] The real and imaginary parts have consistent dimensions and ranges, and are unbiased and have smaller fitting errors than the transmittance-phase output form. By sequentially calculating the response of each metaatom on the metasurface and reshaping it, the complex amplitude response of the entire metasurface to light wave modulation can be obtained. The multi-wavelength dual-polarization deep neural subnetwork replaces the simulation steps based on solving Maxwell's equations with real-time fitting, reducing simulation time and being compatible with deep learning architectures.

[0029] In step 4), the dimensions of the target hologram include three dimensions: wavelength, polarization state, and diffraction distance. The number of wavelengths is n. j , the polarization state is one or two of two orthogonal polarizations, the number is n k , n k = 1 or 2, the number of diffraction distances is n m , where n j ,n m is a natural number ≥ 1, N = n j ×n k ×n m .

[0030] Multi-dimensional multiplexing The target hologram includes three dimensions of wavelength, polarization state and diffraction distance, which is called multi-dimensionality. Simultaneously considering modulation in multiple dimensions of wavelength, polarization state and diffraction distance is called multiplexing.

[0031] The embedding layer includes a multi-layer perceptron (MLP) structure and a convolutional layer. Different wavelengths, polarization states, and diffraction distances correspond to the corresponding coding channels of the embedding layer. Each coding channel corresponds to the third dimension of the internal convolutional layer, that is, the third dimension of the convolutional layer is equal to the number of coding channels, both of which are N. The number of convolution kernels in the convolutional layer is C, and the number of output channels is equal to the number of convolution kernels.

[0032] In the encoder-decoder layer, conventional convolutional layers and corresponding deconvolutional layers are replaced by residual convolutional layers and corresponding residual deconvolutional layers; the encoder-decoder layer consists of multiple layers of residual convolutional layers and corresponding residual deconvolutional layers, and features are transferred through long jump connections; the introduction of residuals enables the model to have more powerful feature encoding and decoding capabilities, and long jump connections ensure that information will not be lost due to scale transformation; and, atrous pyramid pooling (ASPP) is introduced at the bottleneck of the encoder-decoder layer, and multi-scale atrous convolution is realized through convolutions with different expansion rates and global pooling operations, which improves the network's receptive field and helps it capture multi-scale image information to enhance encoding performance.

[0033] In the structure output, the output structure parameter tensor is additionally perturbed with Gaussian noise to simulate the influence of the inevitable line width error in the process of metasurface processing. After the introduction of perturbation, the influence of the processing error can be learned and alleviated by the deep learning metasurface design network, making the designed metasurface more robust against noise. K is the number of parameters of the cross-sectional dimensions. For rectangular nanopillars, there are two cross-sectional dimensions of length and width, K = 2; for circular nanopillars, there are one cross-sectional dimension parameter of radius, K = 1; for elliptical nanopillars, there are two cross-sectional dimensions of major axis and minor axis, K = 2.

[0034] The current hologram is calculated based on the physical model, so that the deep learning metasurface design network can be self-supervised trained. According to the complex amplitude of the outgoing light field obtained by the multi-wavelength dual-polarization deep neural sub-network, a multi-dimensional angular spectrum diffraction algorithm is used to improve the calculation efficiency and obtain the current hologram at different wavelengths. The current hologram has multiple diffraction planes. The angular spectrum diffraction algorithm has the same pixel sampling rate for different wavelengths, thereby ensuring the realization of color holograms. The formula is as follows:

[0035]

[0036] Among them, u k represents the polarization state, λ j represents wavelength, z m represents the diffraction distance, n j 、n k and n m are the wavelength, polarization state and number of diffraction distances, respectively, and Represents the Fourier transform and inverse transform, U out represents the complex amplitude of the light field emitted by the multi-wavelength dual-polarization deep neural sub-network, exp represents the e-exponent, the brackets represent the transfer function of the angular spectrum component, i is the imaginary unit, and f x and f y represents the spatial frequency in the x and y directions, and I represents the intensity distribution of the outgoing light fields with different polarization states at different diffraction distances under the incident light fields with different wavelengths, that is, the current hologram.

[0037] In step 5), the multi-channel composite loss function is a weighted loss function of multiple channels of Pearson loss, multi-scale structural similarity loss, perceptual loss, and peak signal-to-noise ratio loss, and the expression is:

[0038]

[0039] Among them, L nNPCC , L MS-SSIM , L PE and L PSNRThey represent the normalized negative Pearson coefficient, multi-scale structural similarity loss, perceptual loss, and peak signal-to-noise ratio loss, respectively. The horizontal line above represents the average loss in each polarization channel and at different diffraction distances. α and β represent the weights of Pearson loss and multi-scale structural similarity loss, respectively, α+β=1, each varying with wavelength. γ and η are the weights of perceptual loss and peak signal-to-noise ratio loss, respectively, set to 0.05 and 0.1. The form of the loss function ensures the visual similarity between the reconstructed image and the target hologram, thereby improving the quality of the metasurface hologram.

[0040] Advantages of the present invention:

[0041] This paper introduces artificial intelligence into the design process of metasurfaces and proposes a fast design method for metasurfaces based on deep transfer learning. In the design process of multi-dimensional multiplexed metasurfaces, the problems of long calculation time and poor design effect are solved at the same time. Its advantages are reflected in two aspects:

[0042] 1. Compared with the reverse design method with excellent design effect but long design time: the present invention uses deep learning to realize a self-supervised training-testing architecture, thereby giving full play to the dynamic learning ability of artificial intelligence; after training, there is no need to perform time-consuming iterative processes, but to directly reconstruct the metasurface structure parameters according to the determined network weights; for the metasurface design for multi-dimensional multiplexed holograms, the design time of the present invention is less than one second, and the design efficiency is improved by more than 1,000 times;

[0043] 2. Compared with the existing deep learning metasurface design method which takes less time but has poor design effect:

[0044] 1) The network proposed in the present invention introduces residual convolution and dilated pyramid pooling operations in the process of metasurface structure parameter reconstruction to enhance the encoding performance, which greatly improves the design effect of metasurface holograms;

[0045] 2) The present invention introduces a transfer learning method to help the network's functions effectively switch between different application scenarios; when faced with new tasks that change the reuse dimension and the number of channels, the network exhibits faster convergence ability and lower training loss, thereby improving training efficiency;

[0046] 3) The present invention overcomes the shortcoming that the existing deep learning metasurface design methods are limited to monochrome images by introducing a multi-wavelength dual-polarization deep neural sub-network;

[0047] 4) The present invention takes into account the engineering manufacturing problems of metasurfaces: since the process accuracy in micro-nano processing cannot reach 1nm, the metastructure inevitably has line width errors; traditional metasurface designs are severely affected by errors, and the present invention proposes to pre-introduce line width perturbations of the metasurface structure into the network to simulate errors; through this method, the samples designed by the present invention can alleviate the impact caused by process accuracy issues and show stronger robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of an embodiment of a method for rapid design of a multi-dimensional multiplexed holographic metasurface based on deep transfer learning of the present invention;

[0049] Figure 2 A schematic diagram of a superatom three-dimensional structure and a schematic diagram of a multi-wavelength dual-polarization deep neural sub-network of an embodiment of the multi-dimensional multiplexed holographic metasurface rapid design method based on deep transfer learning of the present invention, wherein (a) is a schematic diagram of the three-dimensional structure of the superatom, and (b) is a schematic diagram of the architecture of the multi-wavelength dual-polarization deep neural sub-network;

[0050] Figure 3 A schematic diagram of a deep learning supersurface design network of an embodiment of a multi-dimensional multiplexed holographic supersurface rapid design method based on deep transfer learning of the present invention;

[0051] Figure 4 A schematic diagram of an encoder-decoder layer in a deep learning metasurface design network of an embodiment of a multi-dimensional multiplexed holographic metasurface rapid design method based on deep transfer learning of the present invention;

[0052] Figure 5 This is a rendering of the first embodiment of the method for rapid design of a multi-dimensional multiplexed holographic metasurface of the present invention;

[0053] Figure 6 These are the renderings of Example 2 of the method for rapid design of a multi-dimensional multiplexed holographic metasurface of the present invention, wherein (a) is a multi-dimensional multiplexed metasurface hologram designed using a network after transfer learning, and (b) is a picture of a sample of a real-made metasurface. DETAILED DESCRIPTION

[0054] The present invention will be further described below through specific embodiments in conjunction with the accompanying drawings.

[0055] Embodiment 1

[0056] The multi-dimensional multiplexed holographic metasurface rapid design method based on deep transfer learning of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0057] 1) Designing super atoms:

[0058] The metasurface includes a substrate and a plurality of rectangular nanopillars. The substrate is made of fused quartz and the nanopillars are made of titanium dioxide. The surface of the substrate is discretized into 2040×1536 identical two-dimensionally arranged units. The unit period P is 400nm. A nanopillar is arranged at the center of each unit, that is, there are a total of 2040×1536 nanopillars, that is, L=2040, =W1536. A unit with a nanopillar is called a superatom. The cross-sectional size of the nanopillar is variable, ranging from 100 to 300nm. The height of all nanopillars is fixed at 600nm. By changing the structural parameters of the superatom, that is, the length and width of the cross-sectional size of the nanopillar, that is, K=2, the outgoing light field is changed.

[0059] 2) Construct a training dataset for the multi-wavelength dual-polarization deep neural sub-network:

[0060] The incident light field is incident from the bottom of the substrate to the metasurface, and the meta-atom modulates the incident light field; the finite element analysis is used to calculate the output light field after the incident light field passes through a meta-atom; the structural parameters of the meta-atom are traversed, and multiple different wavelengths and two polarization states of the input light field are traversed, and the two polarization states are perpendicular to each other, and the corresponding output light fields under multiple different wavelengths and two polarization states of the incident light field after passing through the meta-atoms with different structural parameters are obtained as the training data set of the multi-wavelength dual-polarization deep neural sub-network; the polarization states of the incident light field include horizontal polarization H and vertical polarization V, and the wavelengths are 480nm, 532nm, 633nm and 680nm respectively; the finite element analysis simulation data includes the complex amplitude output of four wavelengths of light under two polarizations, that is, the real part (Re 1 ,Re 2 ,…Re 8 )

[0061] and the imaginary part (Im 1 ,Im 2 ,…Im 8 );

[0062] 3) Training multi-wavelength dual-polarization deep neural sub-network:

[0063] like Figure 2 As shown, a multi-wavelength dual-polarization deep neural sub-network is established, with the known structural parameters of the super-atom as input, which are the length and width of the rectangular nanocolumn in this embodiment, and the sampling interval is 5nm. The corresponding outgoing light field after the super-atom under multiple different wavelengths and two polarization states is used as the output, and step 2) is adopted.

[0064] The obtained training data set is used to train the multi-wavelength dual-polarization deep neural sub-network, and the weights of the trained multi-wavelength dual-polarization deep neural sub-network are frozen, that is, in subsequent applications, the weights of the multi-wavelength dual-polarization deep neural sub-network remain unchanged; the multi-wavelength dual-polarization deep neural sub-network includes an input layer, a hidden layer, and an output layer, wherein the number of neurons in the four hidden layers is 256, 512, 512, and 256, respectively; the multi-wavelength dual-polarization deep neural sub-network uses the mean square error (MSE) as the loss function and adopts the adaptive moment estimation (Adam) optimizer for training, and the batch size is set to 32.

[0065] The learning rate is 0.001. During the training process, the data is shuffled to avoid bias, and 10% of the data is randomly selected as the test set.

[0066] After the training is completed, the network training set loss is less than 0.005, and the validation set loss is less than 0.006; In Example 1, the designed color hologram contains three colors of red, green and blue, so the wavelengths of the incident light field are 480nm, 532nm and

[0067] 633nm;

[0068] 4) Construct a deep learning hypersurface design network, such as Figure 3 As shown:

[0069] A multi-dimensional multiplexed target hologram is used as the input of a deep learning metasurface design network, and the structural parameters of each metaatom of the metasurface are used as the output; the multi-dimensional multiplexed target hologram includes three single holograms, that is, N=3, and the number of pixels of each single hologram is 2040×1536, which is equal to the number of discretized units of the metasurface. The number of pixels of the multi-dimensional multiplexed target hologram is 2040×1536×3, and each single hologram corresponds to a coding channel. The number of coding channels of the multi-dimensional multiplexed target hologram is 3, which is the sum of the number of coding channels of the output light field in the three dimensions of wavelength, polarization state and diffraction distance; in this embodiment, n k =1, that is, single polarization, n j =3,n m =1, N=3; the deep learning metasurface design network includes an embedding layer, an encoder-decoder layer, a structure output layer, a multi-wavelength dual-polarization deep neural sub-network, and an angular spectrum diffraction algorithm:

[0070] The embedding layer includes a multi-layer perceptron (MLP) structure and a convolutional layer. Different wavelengths, polarization states, and diffraction distances correspond to the corresponding coding channels of the embedding layer. Each coding channel corresponds to the third dimension of the internal convolutional layer, that is, the third dimension of the convolutional layer is equal to the number of coding channels, which is 3. The number of convolution kernels in the convolutional layer is C, and the number of output channels is equal to the number of convolution kernels.

[0071] Encoder-Decoder Layer The encoder-decoder layer consists of 6 residual convolution layers and corresponding residual deconvolution layers. The number of features in each layer doubles, which are 32, 64, ..., 1024, respectively, and the features are transmitted through long jump connections. The introduction of residuals enables the model to have more powerful feature encoding and decoding capabilities, and long jump connections ensure that information is not lost due to scale transformation. In addition, the atrous pyramid pooling operation (ASPP) is introduced at the bottleneck of the encoder-decoder layer. Multi-scale atrous convolution is realized through convolutions with different expansion rates and global pooling operations, which improves the network's receptive field and helps it capture multi-scale image information to enhance encoding performance, such as Figure 4 As shown;

[0072] Through the structure output layer, all reconstructed features are fitted and a 2040×1536×2 structure parameter tensor is output; the structure parameter tensor represents the structural parameters of each meta-atom of the metasurface; in the structure output layer, Gaussian perturbation noise with a mean of 0 and a standard deviation of 12nm is added to the output structure parameter tensor to simulate the influence of the inevitable line width error in the manufacturing process; after the introduction of the perturbation, the influence of the processing error can be learned and alleviated by the deep learning metasurface design network, making the designed metasurface more robust against noise;

[0073] The multi-wavelength dual-polarization deep neural sub-network calculates the modulation response of each super-atom on the metasurface to the incident light field according to the structural parameters of each super-atom, obtains the outgoing light field at different wavelengths and polarization states, and outputs it in the form of complex amplitude, i.e., real part and imaginary part;

[0074] Using the complex amplitude of the outgoing light field obtained from the multi-wavelength dual-polarization deep neural sub-network, a multi-dimensional angular spectrum diffraction algorithm is used to improve the computational efficiency and obtain the current hologram at different wavelengths. The current hologram has multiple diffraction planes. The angular spectrum diffraction algorithm has the same pixel sampling rate for different wavelengths, thus ensuring the realization of color holograms. The formula is as follows:

[0075]

[0076] Among them, u k represents the polarization state, λ j represents wavelength, z m represents the diffraction distance, and the number of wavelength, polarization state and diffraction distance are n respectively. j 、n k and n m , and Represents the Fourier transform and inverse transform, U out represents the complex amplitude of the light field emitted by the multi-wavelength dual-polarization deep neural sub-network, exp represents the e-index, the brackets represent the transfer function of the angular spectrum component, i is the imaginary unit, and fx and f y represents the spatial frequency in the x-direction and the y-direction, and I represents the intensity distribution of the outgoing light field with different polarization states at different diffraction distances under the incident light of different wavelengths, that is, the dual-polarization color stereo hologram; the pixel sampling rate of angular spectrum diffraction is the same for light of different wavelengths, which ensures the realization of color holographic images;

[0077] 5) Self-supervised training of deep learning hypersurface design network:

[0078] Construct a multi-channel composite loss function, based on the input multi-dimensional multiplexed target hologram and the current hologram calculated in step 4), the network uses the DIV2K database as a training data set, the database contains 900 high-resolution images of natural scenes with complex textures (pixels are set to 2040×1536), of which 800 are used as training sets and 100 are used as test sets; during the training process, a multi-channel composite loss function is used for loss calculation, and the Adam optimizer is used for gradient backpropagation and weight update, the learning rate is 0.0012, and the deep learning super surface design network is trained by self-supervision; the deep learning super surface design network has the characteristics of self-supervision, and there is no need to manually construct labels. Only the multi-dimensional multiplexed target hologram can be input for training to obtain a trained deep learning super surface design network, and the output of the deep learning super surface design network represents the structural parameters of each super atom of the super surface. The 2040×1536×2 structural parameter tensor, that is, the super surface design is completed; after the training is completed, the weight file of the network is extracted, the port corresponding to the encoder-decoder layer is found, and the weights of each node inside it are obtained for the subsequent transfer learning of the second embodiment;

[0079] 6) Applying Deep Learning Hypersurface Design Network Using Transfer Learning:

[0080] After obtaining the trained deep learning metasurface design network, the multi-dimensional multiplexed target hologram is used as input to obtain the structural parameters of each meta-atom on the metasurface, that is, to design the metasurface; after the training of the deep learning metasurface design network is completed, the number of encoding channels is fixed. When faced with the metasurface design task of multi-dimensional multiplexed target holograms with different numbers of encoding channels, rapid switching is performed through transfer learning methods.

[0081] Figure 5 This is the result of the metasurface color holography designed according to the natural scene target after the network of Example 1 is trained. Each row represents a different design, and the average calculation time is 0.7s. R, G, and B represent the holographic results of the three colors of red, green, and blue, respectively, and R+G+B represents a color holographic image. The peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM) are calculated for the color hologram for imaging quality evaluation, and it is proven that the design effect is excellent. The formulas are:

[0082]

[0083]

[0084] The unit of PSNR is decibel (dB), MAX represents the maximum value of all pixels in the reconstructed image, and MSE represents the mean square error between the reconstructed image and the original image. The range of MS-SSIM is 0 to 1, and X and Y represent the reconstructed image and the original image respectively. l, c and s correspond to the brightness, contrast and structural similarity evaluation indicators of the image respectively, and q represents q different evaluation scales obtained by downsampling operation. q , and are the weight factors of the three evaluation indicators, among which a q Does not change with scale, and Varies with the assessment scale.

[0085] Embodiment 2

[0086] In this embodiment, there are two polarization states, four different wavelengths, and two diffraction distances, namely n k =2,n j =4,n m =2, N=16. Migrate the weights of each node extracted in Example 1 to the network of the same architecture and current application scenario, and freeze them. Adjust the dimensions of the embedding layer and output layer before and after the encoder-decoder to make them suitable for the number of encoding channels of the current multi-dimensional multiplexing task. After that, fine-tuning training is performed, using the EMINIST database containing handwritten letters or numbers as the training set, and the learning rate is set to a small 0.0005. After 5 cycles, the training set loss dropped to below 0.33, and the validation set loss dropped to below 0.38. Using direct training, the training loss and validation loss still cannot drop to the same level after 25 cycles, so transfer learning significantly improves the training efficiency.

[0087] Figure 6(a) is the result of the multi-dimensional multiplexed metasurface hologram designed using the network after transfer learning, and the design time is 0.5s. Wherein Sim. represents the theoretical simulation results, and Exp. represents the experimental results. In the figure, every two rows represent a set of diffraction planes, whose distances are 300μm and 800μm respectively; each row represents a different orthogonal polarization, represented by an 'arrow'; each column in the row represents the holographic image corresponding to the incident light of different wavelengths. The network of the present invention introduces error perturbations in advance when calculating the metasurface structure parameters, and the metasurface designed thereafter will have the ability to resist manufacturing errors. The experimental results show that the designed samples have excellent performance, which is consistent with theoretical expectations, and verify that the method of the present invention has excellent metasurface design performance and a fast design time of sub-seconds. The actual manufactured metasurface samples such as Figure 6 (b) as shown.

[0088] Finally, it should be noted that the purpose of publishing the embodiments is to help further understand the present invention, but those skilled in the art can understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be subject to the scope defined in the claims.

Claims

1. A fast design method for multi-dimensional multiplexed holographic metasurfaces based on deep transfer learning, characterized in that: The design method comprises the following steps: 1) Designing super atoms: The metasurface includes a substrate and multiple nanopillars, which discretize the surface of the substrate into L×W identical two-dimensionally arranged units. A nanocolumn is set at the center of each unit, that is, there are L×W nanocolumns in total. A unit with a nanocolumn is called a superatom; the cross-sectional size of the nanocolumn is variable, and the height of all nanocolumns is fixed; the superatom modulates the incident light field, and changes the outgoing light field by changing the structural parameters of the superatom, that is, the cross-sectional size of the nanocolumn; 2) Construct a training dataset for the multi-wavelength dual-polarization deep neural sub-network: The incident light field is incident from the bottom of the substrate to the metasurface, and the meta-atom modulates the incident light field; the finite element analysis is used to calculate the output light field after the incident light field passes through a meta-atom; the structural parameters of the meta-atom are traversed, and multiple different wavelengths and two polarization states of the incident light field are traversed. The two polarization states are perpendicular to each other, and the corresponding output light fields under multiple different wavelengths and two polarization states of the incident light field are obtained after being modulated by meta-atoms with different structural parameters, which are used as the training data set of the multi-wavelength dual-polarization deep neural sub-network; 3) Training multi-wavelength dual-polarization deep neural sub-network: A multi-wavelength dual-polarization deep neural sub-network is established, with the known structural parameters of the meta-atom as input, and the corresponding outgoing light field after the meta-atom modulation under multiple different wavelengths and two polarization states as output. The training data set obtained in step 2) is used to train the multi-wavelength dual-polarization deep neural sub-network, and the weight of the trained multi-wavelength dual-polarization deep neural sub-network is frozen, that is, in subsequent applications, the weight of the multi-wavelength dual-polarization deep neural sub-network remains unchanged; 4) Building a deep learning hypersurface design network: The multi-dimensional multiplexed target hologram is used as the input of the deep learning metasurface design network, and the structural parameters of each meta-atom of the metasurface are used as the output; the multi-dimensional multiplexed target hologram includes N single holograms, and the number of pixels of each single hologram is L×W, which is equal to the number of discretized units of the metasurface. The number of pixels of the multi-dimensional multiplexed target hologram is L×W×N, and a set of wavelengths, polarization states and diffraction distances correspond to a single hologram. Each single hologram corresponds to a coding channel, and N is the number of coding channels of the multi-dimensional multiplexed target hologram, which is the sum of the number of coding channels in the three dimensions of the outgoing light field, namely, wavelength, polarization state, and diffraction distance; The deep learning metasurface design network includes an embedding layer, an encoder-decoder layer, a structure output layer, a multi-wavelength dual-polarization deep neural sub-network, and an angular spectrum diffraction algorithm: The embedding layer encodes the multi-dimensional multiplexed target hologram and converts the multi-dimensional multiplexed target hologram into a computer-recognizable L×W×C feature tensor, which is transmitted to the encoder-decoder layer; where C is the number of output channels of the embedding layer; The encoder-decoder layer extracts, maps and reconstructs the feature tensor, obtains the reconstructed features and passes them to the structure output layer; Through the structure output layer, all the reconstructed features are fitted and the structure parameter tensor of L×W×K is output, where K is the number of parameters of the cross-sectional dimensions; the structure parameter tensor represents the structural parameters of each superatom of the supersurface; The multi-wavelength dual-polarization deep neural sub-network calculates the modulation response of each super-atom on the metasurface to the incident light field according to the structural parameters of each super-atom provided by the structural parameter tensor, obtains the outgoing light field under different wavelengths and polarization states, and outputs it in the form of complex amplitude, that is, the outgoing light field includes real and imaginary parts; The angular spectrum diffraction algorithm is used to obtain the diffraction image of the outgoing light field at different diffraction distances, that is, the current hologram corresponding to the currently designed metasurface is obtained; the pixel sampling rate of angular spectrum diffraction is the same for light of different wavelengths, which ensures the realization of color holographic images; 5) Self-supervised training of deep learning hypersurface design network: Construct a multi-channel composite loss function, based on the input multi-dimensional multiplexed target hologram and the current hologram calculated in step 4), perform loss calculation according to the multi-channel composite loss function, and self-supervise the training of the deep learning metasurface design network; the deep learning metasurface design network has the characteristics of self-supervision, and does not need to manually construct labels. It can be trained by only inputting the multi-dimensional multiplexed target hologram to obtain a trained deep learning metasurface design network. The deep learning metasurface design network outputs a L×W×K structural parameter tensor representing the structural parameters of each metaatom of the metasurface, and the metasurface design is completed; 6) Applying Deep Learning Hypersurface Design Network Using Transfer Learning: After obtaining the trained deep learning metasurface design network, the multi-dimensional multiplexed target hologram is used as input to obtain the structural parameters of each meta-atom on the metasurface, that is, to design the metasurface; after the deep learning metasurface design network training is completed, the number of encoding channels is fixed. When faced with the metasurface design task of multi-dimensional multiplexed target holograms with different numbers of encoding channels, rapid switching is performed through transfer learning methods.

2. The design method according to claim 1, characterized in that: In step 6), for multi-dimensional multiplexed target holograms with different numbers of coding channels, a transfer learning method is used to construct a new deep learning metasurface design network. The same architecture is used in the new deep learning metasurface design network, and the weights corresponding to each node of the encoder-decoder layer of the original deep learning metasurface design network are extracted and migrated to the new deep learning metasurface design network. The weights are frozen, and fine-tuning training is performed in the new deep learning metasurface design network to make the network converge quickly.

3. The design method according to claim 1, characterized in that: In step 1), the substrate is made of transparent material; the nanorods are made of dielectric material.

4. The design method according to claim 1, characterized in that: In step 1), the unit period P is 300nm to 500nm, the cross-sectional dimension should be more than 50nm smaller than the unit period, and the height is fixed at 500nm to 800nm.

5. The design method according to claim 1, characterized in that: In step 2), the step size of the cross-sectional dimensions is 1 to 10 nm; The wavelength of the incident light field is 450 to 800 nm.

6. The design method according to claim 1, characterized in that: In step 4), the dimensions of the target hologram include three dimensions: wavelength, polarization state, and diffraction distance. The number of wavelengths is n. j , the polarization state is one or two of two orthogonal polarizations, the number is n k , n k = 1 or 2, the number of diffraction distances is n m , where n j ,n m is a natural number ≥ 1, N = n j ×n k ×n m .

7. The design method according to claim 1, characterized in that: In step 4), in the structure output layer, Gaussian perturbation noise is additionally added to the output structure parameter tensor to simulate the influence of the inevitable line width error in the process of super surface processing.

8. The design method according to claim 1, characterized in that: In step 4), in the encoder-decoder layer, conventional convolutional layers and corresponding deconvolutional layers are replaced by residual convolutional layers and corresponding residual deconvolutional layers; the encoder-decoder layer is composed of multiple layers of residual convolutional layers and corresponding residual deconvolutional layers, and features are transferred through long jump connections; the introduction of residuals enables the model to have more powerful feature encoding and decoding capabilities, and long jump connections ensure that information will not be lost due to scale transformation; and a dilated pyramid pooling operation is introduced at the bottleneck of the encoder-decoder layer, and multi-scale dilated convolutions are realized through convolutions with different expansion rates and global pooling operations.

9. The design method according to claim 1, characterized in that: In step 4), the angular spectrum diffraction algorithm has the same pixel sampling rate for different wavelengths, and the formula is as follows: Among them, u k represents the polarization state, λ j represents wavelength, z m represents the diffraction distance, n j 、n k and n m are the wavelength, polarization state and number of diffraction distances, respectively, and Represents the Fourier transform and inverse transform, U out represents the complex amplitude of the light field emitted by the multi-wavelength dual-polarization deep neural sub-network, i is an imaginary unit, and f x and f y represents the spatial frequency in the x and y directions, and I represents the intensity distribution of the outgoing light fields with different polarization states at different diffraction distances under the incident light fields with different wavelengths, that is, the current hologram.

10. The design method according to claim 1, characterized in that: In step 5), the multi-channel composite loss function is a weighted loss function of multiple channels of Pearson loss, multi-scale structural similarity loss, perceptual loss, and peak signal-to-noise ratio loss, and the expression is: Among them, L nNPCC , L MS-SSIM , L PE and L PSNR They represent the normalized negative Pearson coefficient, multi-scale structural similarity loss, perceptual loss and peak signal-to-noise ratio loss respectively. The horizontal line above them represents the average loss under each polarization channel and different diffraction distances. α and β represent the weights of Pearson loss and multi-scale structural similarity loss respectively. γ and η are the weights of perceptual loss and peak signal-to-noise ratio loss respectively.

Images