Superlens reverse design method based on selective dual-channel sensing fusion network

Abstract

The invention provides a super-lens reverse design method based on a selective dual-channel perception fusion network, and the method comprises the steps: obtaining a picture-phase spectrum data set through obtaining K pairs of original images and corresponding original phase spectrums; inputting the data to an adaptive difference frequency feature extraction network ADFFNet; inputting a selective non-common feature interaction network SFIFNet to respectively obtain image frequency features and spectral frequency features; the fusion reconstruction features of the image and the fusion reconstruction features of the spectrum are obtained; obtaining a picture generation result and a spectrum generation result; according to the method, the non-common frequency characteristics between the image and the corresponding spectrum can be deeply mined and deeply fused, the mapping accuracy between the image and the corresponding spectrum response can be improved, and the generation accuracy and efficiency of the image and the spectrum can be improved.

Description

Technical Field

[0001] The present invention relates to a superlens reverse design method based on a selective dual-channel perception fusion network, and belongs to the technical field of superlens. Background Art

[0002] A superlens is a micro-lens array composed of nanoscale structures, which uses engineered micro-atoms to determine its own optical response. With the advancement of nanofabrication technology, superlenses have been widely used in holography, beam splitting, optical imaging, sensing, and miniaturized lenses. Therefore, the efficient design of superlenses that perform operations such as polarization modulation, filtering, and holography is crucial in the design of emerging optical devices.

[0003] The design of superlenses is very difficult. Traditionally, the electromagnetic field model of Maxwell's equations is used to describe the underlying physical phenomena of light. However, this model contains a set of coupled partial differential equations, and it is usually difficult to obtain explicit solutions. Therefore, traditional superlens design uses finite element method (FEM) and finite difference time domain method (FDTD) to perform full-wave numerical simulation, and scans parameters multiple times to find the optimal solution. This iterative solution strategy will consume a lot of computing resources and time and fail to obtain different and unique optimal nano-atom designs. Moreover, due to the mismatch of the solution space, when the geometric structure is mapped according to the spectral response, there will be multiple solutions, which makes the inverse design of the geometric structure based on the spectral response usually not feasible.

[0004] In recent years, with the development of neural networks, significant progress has been made in solving the inverse design problem. In 2020, Abhishek Mall et al. proposed a bidirectional encoder (biAE) in the Fast Design of Plasmonic Metasurfaces Enabled by Deep Learning paper, using a Gan network to learn the forward and inverse mapping between the geometric structure and response spectrum of the hypersurface, and optimize in multiple hypersurface topological spaces at the same time, instead of optimizing in one hypersurface topological space as before, thereby solving the one-to-many mapping problem in inverse design. However, this model may lead to overfitting in certain situations, and cause model collapse and significant noise generation, resulting in poor model performance. In 2024, Yuansan Liu et al. proposed a bidirectional adversarial autoencoder (BiAAE) in the Bidirectional Adversarial Autoencoders for the design of Plasmonic Metasurfaces paper, combining the advantages of traditional variational autoencoders in representing information through latent space vectors, while retaining the training method of generative adversarial networks (Gan), which greatly improved the problems of overfitting and noise suppression. However, this adversarial encoder still has the disadvantages of mode collapse and unstable training process, because the game between the generator and the discriminator inside the network may cause the gradient to disappear or explode. In essence, the mapping and generation tasks of image-based meta-atom design and spectral response are actually multimodal fusion and multimodal generation tasks. In 2024, Wenbing Li et al. introduced the selective state transfer network Manba into multimodal tasks for the first time in the paper Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Model. Experimental data show that this network has good performance in forward and reverse generation tasks and multimodal fusion tasks. However, in the existing methods for processing multimodal tasks, the features between modalities are usually only aligned to a unified representation space through mechanisms, but the complementary cross-modal information exchange is not fully utilized, or the internal propagation of a single modality is ignored, and the modal features are aggregated into a whole, which will lead to serious information feature redundancy, information feature loss, and difficulty in alignment supervision during modal interaction.

[0005] The above neural networks all have problems such as poor quality of meta-atom generation parameters and poor fitting results when there are multiple peaks in the spectral curve. In summary, the existing metalens inverse design methods have a lot of room for improvement and exploration. Summary of the invention

[0006] The purpose of the present invention is to provide a superlens inverse design method based on a selective dual-channel perception fusion network to solve the problem in the prior art that the accuracy of image and spectrum generation needs to be improved.

[0007] The technical solution of the present invention is:

[0008] A method for reverse design of a superlens based on a selective dual-channel perception fusion network comprises the following steps:

[0009] S1. Get K pairs of original images I p The corresponding original phase spectrum I s , we get the original image-original phase spectrum pair {I p ,I s} K The constructed image-phase spectrum dataset is divided into a training set and a test set;

[0010] S2, the original image I p The corresponding original phase spectrum I s Input to the adaptive difference frequency feature extraction network ADFFNet to extract high-frequency features F with inter-modal differences s H , low frequency feature F s L .

[0011] S3, the original image I p With the original phase spectrum I s , and the high-frequency feature F obtained in step S2 s H , low frequency feature F s L , input the selective non-common feature interaction network SFIFNet and output the image frequency information F pD and spectral frequency information F sD , corresponding to the original image I p and the original spectrum I s Add together to get the image frequency feature F pDo and the spectral frequency characteristic F sDo ;

[0012] S4, the image frequency feature F pDo and the spectral frequency characteristic F sDo Input the selective multi-dimensional common feature extraction network MMFMamba to obtain the image fusion reconstruction features F D p Fusion with the spectrum to reconstruct the feature F D s ;

[0013] S5, reconstruct the image fusion feature F D p Fusion with the spectrum to reconstruct the feature F D s After image reconstruction and spectral reconstruction are performed in the input modality reconstruction network, the generated image results and generated spectral results are obtained.

[0014] Furthermore, in step S2, the adaptive difference frequency feature extraction network ADFFNet includes a mask self-attention mechanism module, a dual-channel adaptive high- and low-frequency information decomposition module, and a convolutional neural network CNN.

[0015] Encoder network: for the input original image I p and the original phase spectrum I S Perform dimensional transformation to obtain parameters k, q, v;

[0016] Masked self-attention mechanism module: obtain the attention mask feature X from the input parameters k, q, v l ;

[0017] Dual-channel adaptive high- and low-frequency information decomposition module: adaptive attention encoding feature X of input l After the image self-attention feature extraction channel, discrete wavelet transform DWT is performed to obtain the global high-frequency features of the image F s pH and image low-frequency features F s pL , while inputting the adaptive attention encoding feature X l After the spectral self-attention feature extraction channel, the discrete wavelet transform DWT is performed to obtain the spectral global high-frequency features F s sH and the spectral low-frequency feature F s sL ;

[0018] Convolutional neural network CNN: global high-frequency features F of the input image s pH and image low-frequency features F s pL , spectral global high-frequency feature F s sH and the spectral low-frequency feature F s sL After learning the frequency difference features respectively, the high-frequency features F with inter-modal differences are output s H , low frequency feature F s L .

[0019] Furthermore, in step S3, the selective non-common feature interaction network SFIFNet outputs the image frequency information F pD , specifically,

[0020] y plinear =Linear1(LayerNorm(X p ));

[0021] y pm =σ1(CDWConv(y plinear ))

[0022] y pfreq =FS{y pm ,F s H ,F s L}

[0023]

[0024] Among them, X p The original input image I p The encoded image features are obtained by the convolutional neural network encoder CNN Encoder. LayerNorm is the normalization processing layer. Linear1 and Linear2 are linear layer 1 and linear layer 2 respectively. plinear is the image feature output from the linear layer 1, y pm is the image preprocessing feature, CDWConv(·) is a cascaded deep convolution, which is used to capture the depth information and shallow temporal features; σ1(·) and σ2(·) are both activation function operations, y pfreq is the inter-modal image frequency feature, F s H is a high-frequency feature with inter-modal differences, F s L is the low-frequency feature with inter-modal differences, FS is the selectable inter-modal frequency space state transfer model FFISSM; It is Hadamard.

[0025] Furthermore, the inter-modal image frequency feature y can be obtained in the optional inter-modal frequency space state transfer model FFISSM pfreq , specifically,

[0026] The image preprocessing feature y pm Through the linear layer Linear, we get the transformation parameter matrices B', C' and the discrete time matrix △t; the transformation parameter matrix A' is initialized by increasing numbers, that is, from 1 to N 2Assign values ​​incrementally, then take log; according to the zero-order hold method formula, discretize the transformation parameter matrix A`, B` to obtain the time discretization transformation parameter matrix A t ,B t ;

[0027] Get the new state transfer matrix A n , with the new state transfer matrix A n As the core, and the time discretization transformation parameter matrix B t Combined, we get the hidden state matrix h t ′, and then obtain the inter-modal image frequency feature y pfreq :

[0028] A n =A t +αf low +βf high

[0029] h t ′=A n h t +B t y pm

[0030] y pfreq =C`h t ′+IDWT(f low ,f high ).

[0031] Among them, α, β are learnable parameters, and high-frequency enhancement information f high With low frequency enhancement information f low are the high-frequency features of the image F obtained in step S2 s pH and image low-frequency features F s pL After convolution, feature enhancement is performed, and y pm is the image preprocessing feature, h t is the state matrix, and IDWT(·) represents the inverse discrete wavelet transform.

[0032] Furthermore, in step S3, the selective non-common feature interaction network SFIFNet outputs the spectral frequency information F sD , specifically,

[0033] y Slinear =Linear1(LayerNorm(X s ));

[0034] y Sm =σ1(CDWConv(y Slinear ))

[0035] y Sfreq =FS{y Sm ,F s H ,F s L}

[0036]

[0037] Among them, X s is the input original phase spectrum I s The encoded spectral features are obtained by the convolutional neural network encoder CNN Encoder. LayerNorm is the normalization processing layer. Linear1 and Linear2 are linear layer 1 and linear layer 2 respectively. Slinear is the spectral feature of the output of the linear layer 1, y sm is the spectral preprocessing feature, CDWConv(·) is a cascaded deep convolution, which is used to capture the depth information and shallow temporal features; σ1(·) and σ2(·) are both activation function operations, y Sfreq is the inter-modal spectral frequency characteristic, F s H is a high-frequency feature with inter-modal differences, F s L is the low-frequency feature with inter-modal differences, FS is the selectable inter-modal frequency space state transfer model FFISSM; It is Hadamard.

[0038] Further, in step S4, the selective multi-dimensional common feature extraction network MMFMamba includes a first normalization layer LayerNorm1, a multi-branch selective commonality fusion module and a second normalization layer LayerNorm2, and the image frequency feature F pDo and the spectral frequency characteristic F sDo After passing through the corresponding normalized LayerNorm layer, the image normalized feature F is obtained. LN p and the spectral normalization feature F LN s And output to the feature fusion module, the multi-branch selective commonality fusion module selectively fuses the features between the image and spectral modalities to obtain the image fusion features F p I1 , image fusion feature F p I2 , spectral fusion feature F s I3 and spectral fusion feature F s I4 , image fusion feature F p I1And image fusion feature F p I2 After addition, it passes through the second normalization layer LayerNorm2 and then is combined with the image frequency feature F pDo Add together to get the fusion reconstruction feature F of the image D p ; Spectral fusion feature F s I3 and spectral fusion feature F s I4 After addition, it passes through the second normalization layer LayerNorm2 and then is combined with the spectral frequency feature F sDo Add together to get the fusion reconstruction feature F of the spectrum D s .

[0039] Furthermore, in the multi-branch selective commonality fusion module, the image fusion feature F is obtained. p I1 , image fusion feature F p I2 , spectral fusion feature F s I3 and spectral fusion feature F s I4 , specifically:

[0040] F p I1 =Mamba(CDWConv(F LN p ))⊙SiLU(F LN p )

[0041] F p I2 =F2Block(F LN p , F LN s )⊙SiLU(F LN p )

[0042] F s I3 =F2Block(F LN p , F LN s )⊙SiLU(F LN s )

[0043] F s I4 =Mamba(CDWConv(F LN s ))⊙SiLU(FLN s )

[0044] Among them, F LN p is the image normalization feature, F LN s is the spectral normalization feature, CDWConv(·) is the convolution operation, ⊙ is the element-wise multiplication, SiLU(·) is the activation operation, Mamba(·) is the selective state transfer operation, and F2Block(·) is the dual-channel feature selective extraction module.

[0045] Furthermore, the dual-channel feature selective extraction module F2Block includes a dual-channel convolutional neural encoding network, a convolution layer DWConv, an activation function layer SiLU, a selective deep alignment state transfer model AFSSM and a normalization layer LayerNorm.

[0046] Dual-channel convolutional neural encoding network: input image normalization feature F LN p After three layers of convolutional neural network Layer i p , where i = 1, 2, 3, the convolutional neural coding network includes a convolutional layer DWConv, an activation function layer SiLU, and a fully connected layer Dense, which performs dimensional transformation and image feature alignment, and outputs new features as Input spectral normalization feature F LN s After four layers of convolutional neural network Layer i s , where i = 1, 2, 3, 4. After dimensional transformation and image feature alignment, the new features are output as The new feature F s I and the new feature F p I After being sent to the fully connected layer for feature fusion, the fused feature F is obtained sp I ;

[0047] Convolutional layer DWConv: fusion feature F of the input sp I After the convolution operation, the output is sent to the activation function layer SiLU;

[0048] Activation function layer SiLU: used to obtain the fused forward feature F sp I ′ and output to the selective deep alignment state transfer model AFSSM;

[0049] Selective Deep Alignment State Transfer Model AFSSM: Input fusion feature F sp I And fusion forward feature F sp I ′, through the selective state transfer equation, the intermediate state characteristic F is obtained out `And output to the normalization layer LayerNorm;

[0050] Normalization layer LayerNorm: After normalization and dimension reduction, the intermediate feature F is output out .

[0051] Furthermore, in the selective deep alignment state transfer model AFSSM, the forward feature F is fused sp I 'After initialization, the original state transfer matrix A, B, C, D is obtained, and then the input fusion feature F sp I The original state transfer matrices B and C are sent to the linear layer Linear3 and the linear layer Linear4 for hybrid alignment to obtain the fusion state transfer matrices M and N respectively. Finally, the intermediate state feature F is obtained through the selective state transfer equation. out `, the specific process formula is as follows:

[0052] M=Linear3(B,F sp I )

[0053] N=Linear4(C,F sp I )

[0054] h′=Ah t +MF sp I′

[0055] F out `=Nh′+DF sp I′

[0056] Among them, F sp I is the input fusion feature, h t is the state matrix, and h' is the aligned hidden state matrix.

[0057] The beneficial effects of the present invention are:

[0058] 1. This method of reverse design of superlens based on selective dual-channel perception fusion network replaces the traditional superlens design method with a new neural network design method, greatly reducing the calculation and time cost of superlens design. At the same time, the new neural network can deeply mine the non-common frequency characteristics between superlens meta-atoms and corresponding spectra and perform deep fusion, which can increase the accuracy of mapping between superlens meta-atoms and corresponding spectral responses, and realize the accuracy and efficiency of generating superlens meta-atoms and spectra in superlens reverse design.

[0059] 2. The present invention adopts an adaptive differential frequency feature extraction network ADFFNet, which can preliminarily extract the non-common frequency features between the meta-atom images and the corresponding spectra from the implicit space, deeply retain the non-common features between different modalities, and provide feature input for subsequent feature fusion.

[0060] 3. This superlens inverse design method based on the selective dual-channel perception fusion network can deeply fuse non-common features through the selective non-common feature interaction network SFIFNet, which can greatly alleviate the information redundancy and loss problems of feature fusion between multiple modalities, thereby improving the accuracy of subsequent images and spectral responses.

[0061] 4. This superlens inverse design method based on the selective dual-channel perception fusion network not only pays attention to the internal propagation between individual modes but also fully integrates the complementary feature information between modes through the selective multi-dimensional common feature extraction network MMFMamba, which can greatly increase the accuracy of the mapping between meta-atoms and spectral responses. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a flow chart of a method for reverse design of a superlens based on a selective dual-channel sensing fusion network according to an embodiment of the present invention;

[0063] Figure 2 is a schematic diagram illustrating an adaptive difference frequency feature extraction network ADFFNet in an embodiment;

[0064] Figure 3 is a schematic diagram for explaining the selective non-common feature interaction network SFIFNet in the embodiment;

[0065] Figure 4 is a schematic diagram illustrating an inter-modal frequency space state transfer model FFISSM that can be selected in the embodiment;

[0066] Figure 5 is a schematic diagram illustrating the selective multi-dimensional common feature extraction network MMFMamba in the embodiment;

[0067] Figure 6is a schematic diagram for explaining a dual-channel feature selective extraction module F2Block in an embodiment;

[0068] Figure 7 is a schematic diagram illustrating the selective deep alignment state transfer model AFSSM in the embodiment;

[0069] Figure 8 The image result and the original image I generated in the embodiment p A comparative diagram of

[0070] Fig. 9 The spectral result and the original phase spectrum I generated in the embodiment s A comparison diagram of . DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them.

[0072] The embodiment provides a method for reverse design of a superlens based on a selective dual-channel sensing fusion network, such as Figure 1 , including the following steps:

[0073] S1. Get K pairs of original images I P The original phase spectrum I corresponding to the original image S , we get the original image-original phase spectrum pair {I p ,I s} K The image-phase spectrum dataset is composed and divided into a training set and a test set.

[0074] In step S1, the original image The original image can be a metasurface cylindrical image, and the corresponding original phase spectrum Where H, W and C represent the height, width and dimension of the image respectively, and L represents the length of the spectrum.

[0075] S2, the original image I p The corresponding original phase spectrum I s Input to the adaptive difference frequency feature extraction network ADFFNet to extract high-frequency features F with inter-modal differences s H , low frequency feature F s L .

[0076] In step S2, the adaptive difference frequency feature extraction network ADFFNet is as follows Figure 2 :

[0077] 1. First, the original image and the original phase spectrum Transform the dimension in the input encoder network Input Embedding to obtain the corresponding parameters as well as Send the parameters k, q, v to the mask attention encoder MaskAttention to obtain the attention mask feature

[0078] 2. Attention mask feature X l The dual-channel adaptive high- and low-frequency information decomposition module includes the picture self-attention feature extraction channel Picture Sel-Atten and the spectral self-attention feature extraction channel Spec Sel-Atten, and the image self-attention feature X is mined. p l and enhanced spectral self-attention feature X s l ;

[0079] 3. The image self-attention feature X p l and enhanced spectral self-attention feature X s l Discrete wavelet transform DWT is performed to decompose the global high-frequency features F of the image. s pH , image low frequency features F s pL , spectral global high-frequency feature F s sH , spectral low-frequency feature F s sL ;

[0080] 4. The global high-frequency feature F of the image s pH , image low frequency features F s pL , spectral global high-frequency feature F s sH , spectral low-frequency feature F s sL The frequency difference feature learning is performed through a four-layer convolutional neural network CNN, which includes normalization processing LayerNorm, convolution layer Conv, and linear layer Linear. Then the scale is adjusted to ensure that the feature size remains unchanged, and the final output is the high-frequency feature F with inter-modal differences. s H , low frequency feature F s L .

[0081] In step S2, the masked self-attention mechanism module performs preliminary global information extraction on the image and spectrum to extract the adaptive attention encoding feature X l , which can preliminarily distinguish the non-common features between images and spectra. The dual-channel adaptive high- and low-frequency information decomposition module amplifies the non-common features of images and spectra, so that the global information of the image and the global information of the spectrum are fully extracted and the information between the two modalities is not lost. After that, the global information of the image and spectrum is subjected to discrete wavelet transformation respectively to decompose the global high-frequency feature F of the image. s pH and image low frequency information F s pL , spectral global high-frequency feature F s sH and the spectral low-frequency information F s sL , deeply mining features in frequency, such as edge, color and shape features of the image, as well as deep frequency detail information such as spectral inflection points and smoothness. The non-common frequency domain feature fusion module aligns the frequency information of multiple modalities, extracts the differential frequency information between different modalities, and outputs the differential high-low frequency information of image and spectrum fusion.

[0082] S3, the original image I p With the original phase spectrum I s , and the high-frequency feature F with inter-modality differences extracted in step S2 s H , low frequency feature F s L , input the selective non-common feature interaction network SFIFNet and output the image frequency information F pD and spectral frequency information F sD , respectively corresponding to the original image I p and the original spectrum I s Add together to get the image frequency feature F pDo and the spectral frequency characteristic F sDo .

[0083] In step S3, the selective non-common feature interaction network SFIFNet includes a preprocessing module, an optional inter-modal frequency space state transfer model FFISSM, and a feature output module, such as Figure 3 :

[0084] Preprocessing module: input original image I p The original phase spectrum I corresponding to the original image s Perform preliminary feature extraction and alignment encoding CNN Encoder operation to obtain the encoded image feature X pand the encoded spectral feature X s , and then pass through the LayerNorm layer, the linear layer Linear1, the CDWConc convolution layer and the activation layer σ to obtain the image preprocessing feature y pm and spectral preprocessing feature y sm ; Among them, the linear layer Linear1 simultaneously outputs the image feature y plinear With spectral feature y slinear To the feature output module;

[0085] Selectable Inter-modal Frequency Space State Transfer Model FFISSM: The image preprocessing feature X is transformed into p With high frequency features F s H And the low frequency feature F s L , spectral preprocessing feature X s With high frequency features F s H And the low frequency feature F s L , respectively perform feature interaction to obtain the inter-modal image frequency feature y pfreq and the inter-modal spectral frequency characteristics y sfreq ;

[0086] Feature output module: for the input image feature y plinear With spectral feature y slinear After the activation function operation is performed respectively, the corresponding image frequency information F pD and spectral frequency information F sD After performing the Hadamard product and passing through the linear layer Linear2, the image frequency features F pD and the spectral frequency characteristic F sD ,

[0087] In step S3, the selective non-common feature interaction network SFIFNet is as follows Figure 3 , specifically:

[0088] 31) First, the original image Enter the CNN Encoder network, perform preliminary special information extraction and pre-coding operations, and output the encoded image features

[0089] 32) Encode image feature X p After the normalization layer LayerNorm and the linear layer Linear1, the image feature y output by the linear layer 1 is obtained. plinear . After that plinearPerform cascaded deep convolution to extract potential time features; the extracted potential time features are activated by the activation function to obtain the image preprocessing feature y pm ; At the same time, the high-frequency feature F s H And the low frequency feature F s L Joint image preprocessing feature y pm Enter the optional inter-modal frequency space state transfer model FFISSM to obtain the inter-modal image frequency feature y pfreq At the same time plinear After cascaded deep convolution and inter-modal image frequency feature y pfreq Multiply, pass through Linear2 to align the dimensions, and then align with the original image I p Add together and finally get the image frequency feature F pDo Generate F pDo The specific process is as follows:

[0090] y plinear =Linear1(LayerNorm(X p ));

[0091] y pm =σ1(CDWConv(y plinear ))

[0092] y pfreq =FS{y pm ,F s H ,F s L}

[0093]

[0094] F pDo =F pD +I p

[0095] Among them, X p The original input image I p The encoded image features are obtained by the convolutional neural network encoder CNN Encoder. LayerNorm is the normalization processing layer. Linear1 and Linear2 are linear layer 1 and linear layer 2 respectively. plinear is the image feature output from the linear layer 1, y pm is the image preprocessing feature, CDWConv(·) is a cascaded deep convolution, which is used to capture the depth information and shallow temporal features; σ1(·) and σ2(·) are both activation function operations, F s His a high-frequency feature with inter-modal differences, F s L is the low-frequency feature with inter-modal differences, FS is the selectable inter-modal frequency space state transfer model FFISSM; is the Hadamard product. Image frequency feature F pDo Carrying spectral difference information.

[0096] 33) Similarly, the original phase spectrum Send it to the CNN Encoder network for preliminary special diagnosis information extraction and feature mapping encoding operations, and output the encoded spectral features

[0097] 34) Repeat the process of 32) and the final spectral frequency characteristic F sDo Generate F sDo The formula is as follows:

[0098] y Slinear =Linear1(LayerNorm(X s ));

[0099] y Sm =v1(CDWConv(y Slinear ))

[0100] y Sfreq =FS{y Sm ,F s H ,F s L}

[0101]

[0102] F sDo =F sD +I s

[0103] Among them, X s is the input original phase spectrum I s The encoded spectral features are obtained by the convolutional neural network encoder CNN Encoder. LayerNorm is the normalization processing layer. Linear1 and Linear2 are linear layer 1 and linear layer 2 respectively. Slinear is the spectral feature of the output of the linear layer 1, y sm is the spectral preprocessing feature, CDWConv(·) is a cascaded deep convolution, which is used to capture the depth information and shallow temporal features; σ1(·) and σ2(·) are both activation function operations, y Sfreq is the inter-modal spectral frequency characteristic, f s His a high-frequency feature with inter-modal differences, f s L is the low-frequency feature with inter-modal differences, FS is the selectable inter-modal frequency space state transfer model FFISSM; is the Hadamard product, the spectral frequency characteristic F sDo Carry image difference information.

[0104] In step 32) and step 34), the inter-modal frequency space state transfer model FFISSM can be selected to obtain the inter-modal image frequency feature y pfreq and the inter-modal spectral frequency characteristics y sfreq , the process is similar to obtain the inter-modal image frequency feature y pfreq Take the following as an example: Figure 4 :

[0105] 321) The high frequency feature F obtained in step S2 s H And the low frequency feature F s L , after convolution, feature enhancement is performed to obtain high-frequency enhancement information f high With low frequency enhancement information f low ;

[0106] 322) Image preprocessing features The transformation parameter matrix B`∈R is obtained through the linear layer Linear N×1 , C`∈R N×1 And the discrete time matrix △t; for the transformation parameter matrix Use the incremental initialization method, that is, from 1 to N 2 Assign values ​​incrementally, then take log; according to the zero-order hold method formula, discretize the transformation parameter matrix A`, B` to obtain the time discretization transformation parameter matrix A t ,B t .

[0107] 323) By transforming the parameter matrix A based on time discretization t To further optimize, the high frequency enhancement information f high and low frequency enhancement information f low Use the learnable parameters α, β to add to the time discretization transformation parameter matrix A t Among them, the new state transfer matrix A is obtained n . Thus, the high-frequency features of the image with inter-modality differences are realized s pH and image low-frequency features F s pL Able to dynamically adjust the state transfer matrix A t, guiding the state update, and then enabling the high-low frequency features that integrate multi-modal differences to enter the learning of single-modal feature extraction. The specific transformation formula is as follows:

[0108] A n =A t +αf low +βf high

[0109] 324) With the new state transfer matrix A n As the core, and the time discretization transformation parameter matrix B t Combined, we can get the hidden state matrix h t ′:

[0110] h t ′=A n h t +B t y m

[0111] where h t is the state matrix.

[0112] 325) Finally, the high frequency enhancement signal high and low frequency enhancement information f low After the inverse discrete wavelet transform IDWT and the transformation parameter matrix C', the output layer equation is obtained to obtain the output y pfreq , to achieve feature extraction carrying image / spectral differences;

[0113] y pfreq =C`h t ′+IDWT(f low ,f high ).

[0114] Where IDWT(·) stands for inverse discrete wavelet transform.

[0115] S4, the image frequency feature F pDo and the spectral frequency characteristic F sDo Input the selective multi-dimensional common feature extraction network MMFMamba to obtain the image fusion reconstruction features F D p Fusion with the spectrum to reconstruct the feature F D s .

[0116] In step S4, the selective multi-dimensional common feature extraction network MMFMamba includes a first normalization layer LayerNorm1, a multi-branch selective commonality fusion module and a second normalization layer LayerNorm2, and the image frequency feature F carrying spectral difference information pDoand the spectral frequency feature F that carries the image difference information sDo After passing through the corresponding normalized LayerNorm layer, the image normalized feature F is obtained. LN p and the spectral normalization feature F LN s And output to the multi-branch selective commonality fusion module, which selectively fuses the features between the image and spectral modalities to obtain the image fusion features F p I1 , image fusion feature F p I2 , spectral fusion feature F s I3 and spectral fusion feature F s I4 , image fusion feature F p I1 And image fusion feature F p I2 After addition, it passes through the second normalization layer LayerNorm2 and then is combined with the image frequency feature F carrying spectral difference information pDo Add together to get the fusion reconstruction feature F of the image D p ; Spectral fusion feature F s I3 and spectral fusion feature F s I4 After addition, it passes through the second normalization layer LayerNorm2 and then is combined with the spectral frequency feature F that carries the image difference information. sDo Add together to get the fusion reconstruction feature F of the spectrum D s .

[0117] In the multi-branch selective commonality fusion module, the image fusion feature F is obtained p I1 , image fusion feature F p I2 , spectral fusion feature F s I3 and spectral fusion feature F s I4 , specifically:

[0118] F p I1 =Mamba(CDWConv(F LN p ))⊙SiLU(F LN p )

[0119] F p I2=F2Block(F LN p , F LN s )⊙SiLU(F LN p )

[0120] F s I3 =F2Block(F LN p , F LN s )⊙SiLU(F LN s )

[0121] F s I4 =Mamba(CDWConv(F LN s ))⊙SiLU(F LN s )

[0122] Among them, F LN p is the image normalization feature, F LN s is the spectral normalization feature, ⊙ is the element-wise multiplication, CDWConv(·) is a 1x1 convolution operation, SiLU(·) is an activation operation, Mamba9·) is a selective state transfer operation, and F2Block(·) is a dual-channel feature selective extraction module, which is used to align the features between the image and spectral modalities and further integrate the common features of the two.

[0123] The dual-channel feature selective extraction module F2Block includes a dual-channel convolutional neural encoding network, a convolution layer DWConv, an activation function layer SiLU, a selective deep alignment state transfer model AFSSM and a normalization layer LayerNorm.

[0124] Dual-channel convolutional neural encoding network: input image normalization feature F LN p After three layers of convolutional neural network Layer i p , where i = 1, 2, 3. After dimensional transformation and image feature alignment, the new features are output as Input spectral normalization feature F LN s After four layers of convolutional neural network Layer i s , where i = 1, 2, 3, 4. After dimensional transformation and image feature alignment, the new features are output as The new feature F s I and the new feature F p I After being sent to the fully connected layer for feature fusion, the fused feature F is obtained sp I ;

[0125] Convolutional layer DWConv: fusion feature F of the input sp I After the convolution operation, the output is sent to the activation function layer SiLU;

[0126] Activation function layer SiLU: used to obtain the fused forward feature F sp I ′ and output to the selective deep alignment state transfer model AFSSM;

[0127] Selective Deep Alignment State Transfer Model AFSSM: Input fusion feature F sp I And fusion forward feature F sp I ′, through the selective state transfer equation, the intermediate state characteristic F is obtained out `And output to the normalization layer LayerNorm;

[0128] Normalization layer LayerNorm: After normalization and dimension reduction, the intermediate feature F is output out .

[0129] Specifically, in the selective deep alignment state transfer model AFSSM, the forward feature F is fused sp I 'After initialization, the original state transfer matrix A, B, C, D is obtained, and then the input fusion feature F sp I The original state transfer matrices B and C are sent to the linear layer Linear3 and the linear layer Linear4 for hybrid alignment to obtain the fusion state transfer matrices M and N respectively. Finally, the intermediate state feature F is obtained through the selective state transfer equation. out , the specific process formula is as follows:

[0130] M=Linear3(B,F sp I )

[0131] N=Linear4(C,F sp I )

[0132] h′=Ah t +MF sp I′

[0133] F out =Nh′+DF sp I′

[0134] Among them, F sp I is the input fusion feature, h t is the state matrix, and h′ is the aligned hidden state matrix.

[0135] In step S4, the selective multi-dimensional common feature extraction network uses specific modal features to guide the generation of features of another module, aiming to combine local detail features of different modalities, fuse common features, and finally obtain the fusion reconstruction features F of the image. D p Fusion with the spectrum to reconstruct the feature F D s .

[0136] S5, reconstruct the image fusion feature F D p Fusion with the spectrum to reconstruct the feature F D s After image reconstruction and spectral reconstruction are performed in the input modality reconstruction network, the generated image results and generated spectral results are obtained.

[0137] In step S5, the modal reconstruction network includes a selective state transfer output model Mamba and a CNN Decoder network to achieve dimensional reconstruction of the generated meta-atom image and the corresponding spectral response.

[0138] After training with the training set through steps S2 to S5, a trained model is obtained. After parameter optimization with the test set, a final model is obtained. After inputting the image to be generated or the phase spectrum to be generated into the final model, the corresponding generated spectrum or generated picture is obtained.

[0139] Figure 8 The image result and the original image I generated in the embodiment p A comparison diagram of Figure 8 It can be seen that the present invention generates a relatively accurate fitting effect when there are multiple peaks in the spectral curve, which verifies the effectiveness of the present invention.

[0140] Fig. 9 The spectral result and the original phase spectrum I generated in the embodiment s A comparison diagram of Fig. 9 It can be seen that the quality of the meta-atom generation parameters is relatively consistent with the real meta-atom parameters, which verifies the effectiveness of the present invention.

[0141] This superlens inverse design method based on a selective dual-channel perception fusion network can deeply explore the non-common frequency characteristics between images and corresponding spectra and perform deep fusion, which can increase the accuracy of the mapping between images and corresponding spectral responses and improve the accuracy and efficiency of image and spectrum generation.

[0142] This method of reverse design of superlens based on dual-channel extraction network and selective feature fusion mechanism introduces adaptive differential frequency feature extraction network, and uses masked self-attention mechanism module, dual-channel adaptive high-low frequency information decomposition module and non-common frequency domain feature fusion module to preliminarily extract non-common frequency features between meta-atom images and corresponding spectra from implicit space; while introducing an optional inter-modal frequency space state transfer model in the selective non-common feature interaction network to amplify the non-common features of a certain modality of the image / spectrum and extract features, it implicitly adds the non-common features of another module and realizes channel alignment. Introducing multi-branch structure and dual-channel feature selective extraction module in the selective multi-dimensional common feature extraction network not only pays attention to the internal propagation between individual modalities but also fully integrates the complementary feature information between modalities, greatly increasing the accuracy of mapping between meta-atoms and spectral responses. In the modal reconstruction network, the meta-atom image and spectrum are reconstructed and restored using the selective state transfer network.

[0143] This method of reverse design of superlens based on selective dual-channel perception fusion network aims at the problems that reverse design in traditional superlens design is difficult, requires a lot of computing resources and time, and cannot effectively obtain the unique optimal meta-atom design structure in the design. By combining four key networks, namely adaptive difference frequency feature extraction network ADFFNet, selective non-common feature interaction network SFIFNet, selective multi-dimensional common feature extraction network MMFMamba and modal reconstruction network, it can improve the accuracy and efficiency of image and spectrum generation. Among them, the adaptive difference frequency feature extraction network ADFFNet is used to extract the features of the cylindrical image on the surface of the superlens and the corresponding spectral information frequency space, ensuring that the deep invisible features between modules are not lost. The selective non-common feature interaction network SFIFNet is used to implicitly add the non-common features of another module while amplifying the non-common features of a certain modality and extracting features, which greatly alleviates the information redundancy and loss problems of feature fusion between multiple modalities. The selective multi-dimensional common feature extraction network MMFMamba is used for the deep fusion of common features between modalities and fully ensures the feature alignment between modalities, solving the problems of feature misalignment and feature redundancy. The modality reconstruction network is used to generate images and generate reconstructions of spectra.

[0144] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A method for reverse design of a superlens based on a selective dual-channel perception fusion network, characterized in that: The following steps are included: S1. Get K pairs of original images I p The corresponding original phase spectrum I s , we get the original image-original phase spectrum pair {I p ,I s } K The constructed image-phase spectrum dataset is divided into a training set and a test set; S2, the original image I p The corresponding original phase spectrum I s Input to the adaptive difference frequency feature extraction network ADFFNet to extract high-frequency features F with inter-modal differences s H , low frequency feature F s L . S3, the original image I p With the original phase spectrum I s , and the high-frequency feature F obtained in step S2 s H , low frequency feature F s L , input the selective non-common feature interaction network SFIFNet and output the image frequency information F pD and spectral frequency information F sD , respectively corresponding to the original image I p and the original spectrum I s Add together to get the image frequency feature F pDo and the spectral frequency characteristic F sDo ; S4, the image frequency feature F pDo and the spectral frequency characteristic F sDo Input the selective multi-dimensional common feature extraction network MMFMamba to obtain the image fusion reconstruction features F D p Fusion with the spectrum to reconstruct the feature F D s ; S5, reconstruct the image fusion feature F D p Fusion with the spectrum to reconstruct the feature F D s After image reconstruction and spectral reconstruction are performed in the input modality reconstruction network, the generated image results and generated spectral results are obtained.

2. The method for reverse design of a superlens based on a selective dual-channel sensing fusion network according to claim 1, characterized in that: In step S2, the adaptive difference frequency feature extraction network ADFFNet includes a mask self-attention mechanism module, a dual-channel adaptive high- and low-frequency information decomposition module, and a convolutional neural network CNN. Encoder network: for the input original image I p and the original phase spectrum I S Perform dimensional transformation to obtain parameters k, q, v; Masked self-attention mechanism module: obtain the attention mask feature X from the input parameters k, q, v l ; Dual-channel adaptive high- and low-frequency information decomposition module: adaptive attention encoding feature X of input l After the image self-attention feature extraction channel, discrete wavelet transform DWT is performed to obtain the global high-frequency features of the image F s pH and image low-frequency features F s pL , while inputting the adaptive attention encoding feature X l After the spectral self-attention feature extraction channel, the discrete wavelet transform DWT is performed to obtain the spectral global high-frequency features F s sH and the spectral low-frequency feature F s sL ; Convolutional neural network CNN: global high-frequency features F of the input image s pH and image low-frequency features F s pL , spectral global high-frequency feature F s sH and the spectral low-frequency feature F s sL After learning the frequency difference features respectively, the high-frequency features F with inter-modal differences are output s H , low frequency feature F s L .

3. The method for reverse design of a superlens based on a selective dual-channel sensing fusion network according to claim 1, characterized in that: In step S3, the selective non-common feature interaction network SFIFNet outputs the image frequency information F pD , specifically, y plinear =Linear1(LayerNorm(X p )); y pm =σ1(CDWConv(y plinear )) y pfreq =FS{y pm ,F s H ,F s L } Among them, X p The original input image I p The encoded image features are obtained by the convolutional neural network encoder CNN Encoder. LayerNorm is the normalization processing layer. Linear1 and Linear2 are linear layer 1 and linear layer 2 respectively. plinear is the image feature output from the linear layer 1, y pm is the image preprocessing feature, CDWConv(·) is a cascaded deep convolution, which is used to capture the depth information and shallow temporal features; σ1(·, σ2(·) are both activation function operations, y pfreq is the inter-modal image frequency feature, F s H is a high-frequency feature with inter-modal differences, F s L is the low-frequency feature with inter-modal differences, FS is the selectable inter-modal frequency space state transfer model FFISSM; It is Hadamard.

4. The method for reverse design of a superlens based on a selective dual-channel sensing fusion network as claimed in claim 3, characterized in that: In the optional inter-modal frequency space state transfer model FFISSM, the inter-modal image frequency feature y is obtained pfreq , specifically, The image preprocessing feature y pm Through the linear layer Linear, we get the transformation parameter matrices B', C' and the discrete time matrix △t; the transformation parameter matrix A' is initialized by increasing numbers, that is, from 1 to N 2 Assign values ​​incrementally, then take log; according to the zero-order hold method formula, discretize the transformation parameter matrix A`, B` to obtain the time discretization transformation parameter matrix A t ,B t ; Get the new state transfer matrix A n , with the new state transfer matrix A n As the core, and the time discretization transformation parameter matrix B t Combined, we get the hidden state matrix h t ′, and then obtain the inter-modal image frequency feature y pfreq : A n =A t +αf low +βf high h t '=A n h t +B t y pm y pfreq =C`h t ′+IDWT(f low ,f high )。 Among them, α, β are learnable parameters, and high-frequency enhancement information f high With low frequency enhancement information f low are the high-frequency features of the image F obtained in step S2 s pH and image low-frequency features F s pL After convolution, feature enhancement is obtained, y pm is the image preprocessing feature, h t is the state matrix, and IDWT(·) represents the inverse discrete wavelet transform.

5. The method for reverse design of a superlens based on a selective dual-channel sensing fusion network as claimed in claim 3, characterized in that: In step S3, the selective non-common feature interaction network SFIFNet outputs the spectral frequency information F sD , specifically, y Slinear =Linear1(LayerNorm(X s )); y Sm =σ1(CDWConv(y Slinear )) y Sfreq =FS{y Sm ,F H s ,F s L } Among them, X s is the input original phase spectrum I s The encoded spectral features are obtained by the convolutional neural network encoder CNN Encoder. LayerNorm is the normalization processing layer. Linear1 and Linear2 are linear layer 1 and linear layer 2 respectively. Slinear is the spectral feature of the output of the linear layer 1, y sm is the spectral preprocessing feature, CDWConv(·) is a cascaded deep convolution, which is used to capture the depth information and shallow temporal features; σ1(·) and σ2(·) are both activation function operations, y Sfreq is the inter-modal spectral frequency characteristic, F s H is a high-frequency feature with inter-modal differences, F s L is the low-frequency feature with inter-modal differences, FS is the selectable inter-modal frequency space state transfer model FFISSM; It is Hadamard.

6. The method for reverse design of a superlens based on a selective dual-channel sensing fusion network according to any one of claims 1 to 5, characterized in that: In step S4, the selective multi-dimensional common feature extraction network MMFMamba includes a first normalization layer LayerNorm1, a multi-branch selective commonality fusion module and a second normalization layer LayerNorm2, and the image frequency feature F pDo and the spectral frequency characteristic F sDo After passing through the corresponding normalized LayerNorm layer, the image normalized feature F is obtained. LN p and the spectral normalization feature F LN s And output to the feature fusion module, the multi-branch selective commonality fusion module selectively fuses the features between the image and spectral modalities to obtain the image fusion features F p I1 , image fusion feature F p I2 , spectral fusion feature F s I3 and spectral fusion feature F s I4 , image fusion feature F p I1 And image fusion feature F p I2 After addition, it passes through the second normalization layer LayerNorm2 and then is combined with the image frequency feature F pDo Add together to get the fusion reconstruction feature F of the image D p ; Spectral fusion feature F s I3 and spectral fusion feature F s I4 After addition, it passes through the second normalization layer LayerNorm2 and then is combined with the spectral frequency feature F sDo Add together to get the fusion reconstruction feature F of the spectrum D s .

7. The method for reverse design of a superlens based on a selective dual-channel sensing fusion network according to claim 6, characterized in that: In the multi-branch selective commonality fusion module, the image fusion feature F is obtained p I1 , image fusion feature F p I2 , spectral fusion feature F s I3 and spectral fusion feature F s I4 , specifically: F p I1 =Mamba(CDWConv(F LN p ))⊙SiLU(F LN p ) F p I2 =F2Block(F LN p ,F LN s )⊙SiLU(F LN p ) F s I3 =F2Block(F LN p ,F LN s )⊙SiLU(F LN s ) F s I4 =Mamba(CDWConv(F LN s ))⊙SiLU(F LN s ) Among them, F LN p is the image normalization feature, F LN s is the spectral normalization feature, CDWConv(·) is the convolution operation, ⊙ is the element-wise multiplication, SiLU(·) is the activation operation, Mamba(·) is the selective state transfer operation, and F2Block(·) is the dual-channel feature selective extraction module.

8. The method for reverse design of a superlens based on a selective dual-channel sensing fusion network according to claim 6, characterized in that: The dual-channel feature selective extraction module F2Block includes a dual-channel convolutional neural encoding network, a convolution layer DWConv, an activation function layer SiLU, a selective deep alignment state transfer model AFSSM and a normalization layer LayerNorm. Dual-channel convolutional neural encoding network: input image normalization feature F LN p After three layers of convolutional neural network Layer i p , where i = 1, 2, 3, the convolutional neural coding network includes a convolutional layer DWConv, an activation function layer SiLU, and a fully connected layer Dense, which performs dimensional transformation and image feature alignment, and outputs new features as Input spectral normalization feature F LN s After four layers of convolutional neural network Layer i s , where i = 1, 2, 3, 4. After dimensional transformation and image feature alignment, the new features are output as The new feature F s I and the new feature F p I After being sent to the fully connected layer for feature fusion, the fused feature F is obtained sp I ; Convolutional layer DWConv: fusion feature F of the input sp I After the convolution operation, the output is sent to the activation function layer SiLU; Activation function layer SiLU: used to obtain the fused forward feature F sp I ′ and output to the selective deep alignment state transfer model AFSSM; Selective Deep Alignment State Transfer Model AFSSM: Input fusion feature F sp I And fusion forward feature F sp I ′, through the selective state transfer equation, the intermediate state characteristic F is obtained out `And output to the normalization layer LayerNorm; Normalization layer LayerNorm: After normalization and dimension reduction, the intermediate feature F is output out .

9. The method for reverse design of a superlens based on a selective dual-channel sensing fusion network according to claim 8, characterized in that: In the selective deep alignment state transfer model AFSSM, the forward feature F is fused sp I 'After initialization, the original state transfer matrix A, B, C, D is obtained, and then the input fusion feature F sp I The original state transfer matrices B and C are sent to the linear layer Linear3 and the linear layer Linear4 for hybrid alignment to obtain the fusion state transfer matrices M and N respectively. Finally, the intermediate state feature F is obtained through the selective state transfer equation. out `, the specific process formula is as follows: M=Linear3(B,F sp I ) N=Linear4(C,F sp I ) h′=Ah t +MF sp I′ F out `=Nh′+DF sp I ′ Among them, F sp I is the input fusion feature, h t is the state matrix, and h' is the aligned hidden state matrix.

Images