Chiral metasurface structure parameter optimization model training method, intelligent agent and device

Abstract

The application provides a chiral metasurface structure parameter optimization model training method, an intelligent agent and a device. The method comprises the following steps: training a structure parameter optimization model of a chiral metasurface based on a proximal policy optimization (PPO) algorithm in a reinforcement learning algorithm; sending an action output by the structure parameter optimization model to an environment in a Markov decision system, so that a forward prediction neural network in the environment outputs a spectral response; calculating a reward according to the spectral response; and receiving a next observation state and a new reward to iteratively train the structure parameter optimization model again. The application can effectively improve the response speed of the interactive environment while ensuring the optimization accuracy of the structure parameters of the chiral metasurface, thereby improving the efficiency of the intelligent agent in collecting the required training data, speeding up the training of the intelligent agent, and further effectively improving the optimization efficiency of the structure parameters of the chiral metasurface.

Description

Technical Field

[0001] This application relates to the field of chiral metasurface technology, and in particular to training methods, intelligent agents, and devices for optimizing chiral metasurface structural parameters. Background Technology

[0002] Chirality is widespread in nature and has applications in life sciences, sensing, imaging, and biomedicine. The chiral optical responses of naturally occurring chiral molecules are typically weak and difficult to detect, thus necessitating the design of artificial chiral structures to enhance their optical responses. Chiral metasurfaces have attracted widespread attention due to their ability to arbitrarily adjust and enhance optical responses, and can be used to strengthen the response intensity of chiral molecules. How to efficiently and accurately optimize metasurfaces to achieve the desired chiral response has become a key focus in the design process of chiral metasurfaces.

[0003] Currently, besides brute-force solutions such as manual search and parameter scanning, intelligent algorithms for optimizing the structural parameters of chiral metasurfaces mainly fall into two categories: one combines electromagnetic simulation with optimization algorithms, such as genetic algorithms (GA), simulated annealing (SA), and particle swarm optimization (PSO). These methods are simple and direct in designing chiral metasurfaces, but the complexity of the optimization increases exponentially as the number of parameters to be designed increases, thus affecting the optimization efficiency of chiral metasurfaces. The other type of intelligent algorithm uses neural networks to reverse-engineer chiral metasurfaces. It uses collected structural parameters and corresponding spectral responses as training data to train the neural network. The trained neural network can output the corresponding structural parameters based on the input spectrum. However, during the training process of the reverse-engineered network, there is often a problem that one spectral response may correspond to multiple structural parameters, which increases the difficulty, time consumption, and accuracy of training.

[0004] Therefore, how to improve the efficiency of structural parameter optimization for chiral metasurfaces while ensuring the accuracy of the optimization is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of this application provide a training method, intelligent agent, and device for optimizing chiral metasurface structure parameters, in order to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of this application provides a method for training a chiral metasurface structure parameter optimization model, comprising:

[0007] Based on the Proximal Policy Optimization (PPO) algorithm in reinforcement learning, a structural parameter optimization model for a chiral metasurface is trained using the current observation state and reward, so that the structural parameter optimization model outputs a corresponding action. The observation state data includes the structural parameters of the chiral metasurface, and the action includes the change in the structural parameters.

[0008] The action is sent to an environment that is in the same Markov decision system as itself, so that the forward prediction neural network in the environment outputs the spectral response corresponding to the action, and a new reward corresponding to the action is generated based on the spectral response.

[0009] The system receives the next observation state and the new reward sent by the environment, and then iteratively trains the structure parameter optimization model based on the next observation state and the new reward.

[0010] In some embodiments of this application, the structural parameter optimization model of the chiral metasurface includes: a first neural network and a second neural network;

[0011] The first neural network is used to output the corresponding action based on the observed state in the input;

[0012] The second neural network is used to output a corresponding advantage estimation result based on the observed state in the input, and to update the first neural network by applying the advantage estimation result in a gradient ascent manner.

[0013] In some embodiments of this application, the forward prediction neural network includes: a deep neural network for performing forward prediction;

[0014] The deep neural network is used to output a corresponding spectral response based on the input structural parameters, the spectral response including a circular dichroism (CD) spectrum.

[0015] In some embodiments of this application, before sending the action to an environment that is in the same Markov decision system as itself, the method further includes:

[0016] Obtain the structural parameters and corresponding spectral response labels of the chiral metasurface to obtain the corresponding training set;

[0017] The training set is used to train a deep neural network for forward prediction, so that the deep neural network outputs the spectral response corresponding to the structural parameters. The objective function of the deep neural network is solved based on the spectral response and the spectral response label. The weights of the deep neural network are updated based on the Adam gradient descent method using the corresponding solution results.

[0018] In some embodiments of this application, the observation state data and the training set further include: flag bits;

[0019] The flag bit is used to indicate whether the structural parameters of the uniquely corresponding chiral metasurface are left-handed or right-handed.

[0020] In some embodiments of this application, obtaining the structural parameters and corresponding spectral response labels of the chiral metasurface includes:

[0021] The structural parameters and corresponding spectral response labels of chiral metasurfaces are collected based on the finite-difference time-domain algorithm.

[0022] Another aspect of this application provides a PPO agent, comprising:

[0023] An action generation module is used to train a structural parameter optimization model for a chiral metasurface based on the proximal policy optimization (PPO) algorithm in reinforcement learning. The model is trained using the current observation state and reward, so that the structural parameter optimization model outputs a corresponding action. The observation state data includes the structural parameters of the chiral metasurface, and the action includes the change in the structural parameters.

[0024] An action sending module is used to send the action to an environment that is in the same Markov decision system as itself, so that the forward prediction neural network in the environment outputs the spectral response corresponding to the action and generates a new reward corresponding to the action based on the spectral response.

[0025] An experience receiving module is used to receive the next observation state and the new reward sent by the environment, so as to iteratively train the structure parameter optimization model again based on the next observation state and the new reward.

[0026] Another aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the chiral metasurface structure parameter optimization model training method.

[0027] Another aspect of this application provides a Markov decision system comprising: an interactive PPO agent and an environment;

[0028] The PPO agent is used to execute the chiral metasurface structure parameter optimization model training method described above.

[0029] The environment is equipped with the feedforward prediction neural network, which outputs the spectral response corresponding to the action sent by the PPO agent, and generates a new reward corresponding to the action based on the spectral response.

[0030] Another aspect of this application provides a micro / nano device based on a chiral metasurface, including a glass substrate and a double-layer chiral metal structure located on the surface of the glass substrate, wherein the structural parameters of the double-layer chiral metal structure are set according to the optimization results corresponding to the structural parameters output by the structural parameter optimization model of the chiral metasurface.

[0031] The structural parameter optimization model of the chiral metasurface is pre-trained based on the aforementioned chiral metasurface structural parameter optimization model training method.

[0032] The method for training a chiral metasurface structure parameter optimization model provided in this application is based on the Proximal Policy Optimization (PPO) algorithm in reinforcement learning. It applies the current observation state and reward to train the chiral metasurface structure parameter optimization model, enabling the model to output corresponding actions. The observation state data includes the structure parameters of the chiral metasurface, and the actions include changes in the structure parameters. The actions are then sent to an environment within the same Markov decision system, causing the feedforward prediction neural network in that environment to output the spectral response corresponding to the action and generate a new reward based on that response. The method receives the next observation state and the new reward from the environment and iteratively trains the structure parameter optimization model again based on the next observation state and the new reward. This method utilizes reinforcement learning algorithms to train the chiral metasurface structure parameters. Intelligent optimization is performed by combining the proximal policy optimization algorithm in reinforcement learning with the feedforward prediction neural network. The reverse design process is modeled as a decision problem, satisfying a Markov decision process. Through interaction with the environment, the agent can learn an efficient optimization strategy. This strategy automatically searches for the optimal solution in the parameter space based on the optimization objective, thus effectively ensuring the accuracy of the structural parameters optimization of chiral metasurfaces. By incorporating the feedforward prediction network into the interactive environment, the response speed of the interactive environment can be greatly improved, thereby increasing the efficiency of the agent in collecting the data required for training, accelerating the training of the agent, and improving the accuracy and efficiency of the structural parameter optimization of chiral metasurfaces. This also enhances the reliability and effectiveness of applications of micro / nano devices based on chiral metasurfaces designed using the optimized structural parameters output by the chiral metasurface structural parameter optimization model.

[0033] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon reading the following text, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained through the structures specifically pointed out in the specification and drawings.

[0034] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0035] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings:

[0036] Figure 1(a) is a schematic diagram of the unit cell structure and the definition of circular dichroism (CD) in a three-dimensional chiral metasurface.

[0037] Figure 1(b) is a schematic diagram comparing the left-handed and right-handed enantiomers in the unit cell of a three-dimensional chiral metasurface.

[0038] Figure 2(a) is a schematic diagram of the Markov decision process of the interaction between the agent and the environment.

[0039] Figure 2(b) is a schematic diagram of a feedforward predictive neural network as part of an agent's interactive environment.

[0040] Figure 3 This is a schematic diagram of the overall process of training a chiral metasurface structure parameter optimization model in one embodiment of this application.

[0041] Figure 4 This is a schematic diagram illustrating a specific process of training a chiral metasurface structure parameter optimization model according to an embodiment of this application.

[0042] Figure 5 This is a schematic diagram of a feedforward predictive neural network for a chiral metasurface in one embodiment of this application.

[0043] Figure 6 This is a schematic diagram of the structure of the intelligent agent in another embodiment.

[0044] Figure 7 A schematic diagram of the intelligent design process for chiral metasurfaces based on reinforcement learning, provided as an application example of this application.

[0045] Figure 8 A schematic diagram of the design of a chiral metasurface by combining the PPO reinforcement learning algorithm with a feedforward predictive neural network, which is provided as an application example of this application.

[0046] Figure 9A schematic diagram illustrating the change in reward received by PPO in each round of the training process, provided as an application example of this application.

[0047] Figure 10(a) is a schematic diagram of the design of a trained agent with a target wavelength of 1100nm for designing a left-handed structure, provided in the application example of this application.

[0048] Figure 10(b) is a schematic diagram of the design of a trained agent with a target wavelength of 1035 nm for designing a left-handed structure, provided in the application example of this application.

[0049] Figure 10(c) is a schematic diagram of the design of a trained agent with a target wavelength of 1000 nm for designing a left-handed structure, provided in the application example of this application.

[0050] Figure 10(d) is a schematic diagram of the design of a trained agent with a target wavelength of 930 nm for designing a left-handed structure, provided in the application example of this application.

[0051] Figure 11(a) is a schematic diagram of the design of a trained agent with a target wavelength of 1100nm for designing a right-handed structure, provided in the application example of this application.

[0052] Figure 11(b) is a schematic diagram of the design of a trained agent with a target wavelength of 1035 nm for designing a right-handed structure, provided in the application example of this application.

[0053] Figure 11(c) is a schematic diagram of the design of a trained agent with a target wavelength of 1000 nm for designing a right-handed structure, provided in the application example of this application.

[0054] Figure 11(d) is a schematic diagram of the design of a trained agent with a target wavelength of 930 nm for designing a right-handed structure, provided in the application example of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0056] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0057] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0058] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0059] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0060] Reinforcement learning algorithms represent a novel approach to the inverse design of chiral metasurfaces. By interacting with the environment, they learn corresponding policies for efficient design of chiral metasurfaces. These learned policies are often more efficient than heuristic algorithms, making them suitable for scenarios with frequent design requirements. Combining reinforcement learning with a forward prediction network addresses the issue of slow data collection by the agent, thus improving training efficiency.

[0061] This application relates to the structural parameter optimization problem of chiral metasurfaces and proposes an intelligent design method for chiral metasurfaces based on reinforcement learning algorithms. This method combines the proximal policy optimization algorithm in reinforcement learning algorithms with the forward prediction method based on deep neural networks, and models the reverse design process as a decision problem, so that it satisfies the Markov decision process.

[0062] By interacting with its environment, the agent can learn an efficient optimization strategy that automatically searches for the optimal solution in the parameter space based on the optimization objective. Integrating the positive prediction network as part of the interactive environment significantly improves the environment's response speed, thereby increasing the agent's efficiency in collecting training data and accelerating training. This algorithm can obtain results that satisfy the optimization objective in a very short time (no more than 10 seconds).

[0063] This application considers the design of structural parameters for chiral metasurfaces. The metasurface to be optimized is a periodic bilayer structure based on the Born-Kuhn model, with gold nanorods stacked at both ends in the upper and lower layers. It has C4 rotational symmetry, which can avoid linear birefringence and thus avoid unnecessary polarization conversion.

[0064] In this scenario, a periodic structure of the metasurface is shown in Figure 1(a). When left-handed circularly polarized light (LCP) and right-handed circularly polarized light (RCP) are incident on the periodic structure, the structure absorbs the two types of circularly polarized light differently, resulting in different transmittances of the emitted circularly polarized light. This is called circular dichroism (CD), which is defined as follows:

[0065] CD = T LCP -T RCP (1)

[0066] Among them, T LCP T represents the transmittance of left-handed circularly polarized light. RCP This represents the transmittance when right-handed circularly polarized light is incident.

[0067] Figure 1(b) shows the left-handed enantiomer LH and the right-handed enantiomer RH of the chiral structure. In this scenario, a total of 5 structural parameters were optimized, including the distance D between the upper and lower gold nanorods, the length L of the nanorods, the width W of the nanorods, the gap G between the left and right gold nanorods, and the structural period P.

[0068] The optimization strategy for this structure is learned using a reinforcement learning algorithm.

[0069] The reverse design problem of chiral metasurfaces can be mathematically modeled as an optimization problem, and an optimal solution can be found using optimization algorithms. Optimization algorithms such as genetic algorithms, particle swarm optimization, and simulated annealing can find optimal solutions for specific objectives; however, the strategies used in these algorithms are artificially derived from natural laws, and their efficiency in applying them to specific problems still needs improvement. Many strategies in nature cannot be artificially derived, and they are often more efficient for specific problems. Neural networks can learn the correspondence between spectral responses and structural parameters from collected data, but besides being unsuitable for optimization problems, they have high requirements for the quantity and rationality of the data. Therefore, how to find an efficient strategy to obtain an optimal solution in the shortest time for complex problems and huge parameter spaces is the problem this approach aims to solve. Common optimization algorithms, after obtaining an optimization result based on the objective, require repeating the optimization process in the same amount of time when trying to optimize for other objectives. How to obtain a strategy that can be reused and applied to multiple tasks is another problem this approach aims to solve.

[0070] First, we introduce chiral metasurfaces and their optimized modeling as follows:

[0071] The chiral metasurface selected in this scheme has a strong chiral response, and the interaction between light and matter in the chiral metasurface can be analyzed using a simple harmonic oscillator model. This is of great significance for effective structural parameter design and for a deeper understanding of the interaction between chiral metasurfaces and chiral molecules in practical applications such as biosensing. The metasurface is modeled and simulated using a finite-difference time-domain algorithm. The gold nanorods are covered by a layer of dielectric material with a refractive index of 1.3. The dielectric constant of the gold nanorods is obtained based on measurements by Johnson and Christy. The bottom layer is a glass substrate. According to the actual situation, the range of optimized structural parameters is set as shown in Table 1. The optimization accuracy is selected as 10 nm. In order to avoid conflict between the period and the structure, i.e., the size of the unit structure does not exceed the period set in the simulation, this scheme introduces a period-related parameter Q∈[20,80] to calculate the period. The algorithm optimizes this parameter instead of optimizing the period. The period of the structure is calculated by formula (2).

[0072] P = 2(L + Q) + G (2)

[0073] Table 1. Optimization range of chiral metasurface structure parameters

[0074] D(nm) L(nm) W(nm) G(nm) P(nm) Minimum value 20 100 30 20 260 Maximum value 70 230 90 80 700

[0075] The process of finding an optimal optimization strategy can be modeled as follows:

[0076]

[0077] By interacting with an environment that can respond according to structural parameters through reinforcement learning algorithms, a strategy that can achieve a specific goal in a certain band is obtained. In the design of the chiral metasurface provided in this application, the goal can be to maximize the absolute value of CD at a specified wavelength in a certain parameter space.

[0078] This application utilizes reinforcement learning algorithms to learn efficient strategies for the optimal design of chiral metasurfaces. The reinforcement learning algorithm mainly consists of two parts: an agent and an environment, as shown in Figure 2(a). The environment has a significant impact on the learning effect and efficiency of the agent because it needs to receive the actions performed by the agent and provide feedback on the results. In our problem, the finite-difference time-domain (FDTD) algorithm can be used as part of the environment. However, verifying a structure using FDTD is often time-consuming, significantly reducing the learning efficiency of the agent. To improve the efficiency of training the agent, a deep neural network (DNN), i.e., a feedforward predictive neural network, is used instead of FDTD as part of the environment, as shown in Figure 2(b).

[0079] Based on this, embodiments of this application provide a method for training a chiral metasurface structure parameter optimization model, see [link to relevant documentation]. Figure 3The training method for optimizing the structure parameters of the chiral metasurface specifically includes the following:

[0080] Step 100: Based on the Proximal Policy Optimization (PPO) algorithm in reinforcement learning, train the structural parameter optimization model of the chiral metasurface using the current observation state and reward, so that the structural parameter optimization model outputs the corresponding action. The observation state data includes the structural parameters of the chiral metasurface, and the action includes the change in the structural parameters.

[0081] In step 100, this application uses the Proximal Policy Optimization (PPO) algorithm in reinforcement learning to learn the optimization policy. The PPO reinforcement learning algorithm is developed based on Tensorflow. The reinforcement learning algorithm uses a forward prediction network as part of the environment. This network is developed based on PyTorch. The data required to train this forward network and the corresponding labels are collected using the Finite-Difference Time-Domain (FDTD) algorithm. In order to increase the efficiency of data collection, the lumopt toolkit is used to interact with the FDTD algorithm to automatically generate and collect the dataset.

[0082] Step 200: Send the action to an environment that is in the same Markov decision system as itself, so that the forward prediction neural network in the environment outputs the spectral response corresponding to the action, and generates a new reward corresponding to the action based on the spectral response.

[0083] Step 300: Receive the next observation state and the new reward sent by the environment, and iteratively train the structure parameter optimization model again based on the next observation state and the new reward.

[0084] Specifically, the lumopt Python toolkit can be used to interact with the Finite-Difference Time-Domain (FDTD) algorithm, automatically collecting parameters and corresponding CD spectra of left-handed and right-handed structures. Then, flags are added to the structural parameters of the two chiral structures for differentiation, and the two datasets are mixed, maintaining the correspondence between the spectral labels and structural parameters. A feedforward predictive neural network is built in PyTorch and trained using the collected and processed dataset. The PPO algorithm is built in TensorFlow, using the feedforward predictive neural network as part of the interactive environment. The agent interacts with the environment and learns inverse design policies. The learned agent executes the policy, interacts with the environment based on the feedforward predictive neural network, and quickly finds optimal solutions.

[0085] As described above, the chiral metasurface structure parameter optimization model training method provided in this application intelligently optimizes the structural parameters of chiral metasurfaces based on reinforcement learning algorithms. By combining the proximal policy optimization algorithm and the feedforward prediction neural network within the reinforcement learning algorithm, the reverse design process is modeled as a decision problem, satisfying a Markov decision process. Through interaction with the environment, the agent can learn an efficient optimization strategy. This strategy automatically searches for the optimal solution in the parameter space based on the optimization objective, effectively ensuring the accuracy of the chiral metasurface structure parameter optimization. By incorporating the feedforward prediction network as part of the interactive environment, the response speed of the interactive environment can be greatly improved, thereby increasing the efficiency of the agent in collecting the data required for training, accelerating the agent's training, and thus improving the accuracy and efficiency of chiral metasurface structure parameter optimization. Furthermore, it can enhance the reliability and effectiveness of applications of chiral metasurface-based micro / nano devices designed using the optimized structural parameters output by the chiral metasurface structure parameter optimization model. The optimization method provided in this application can obtain results that meet the optimization objective in a very short time (no more than 10 seconds).

[0086] To improve the reliability and effectiveness of the structural parameter optimization model and further ensure the accuracy of the optimization of the chiral metasurface structural parameters, in the chiral metasurface structural parameter optimization model training method provided in this application embodiment, the chiral metasurface structural parameter optimization model includes: a first neural network and a second neural network;

[0087] The first neural network is used to output the corresponding action based on the observed state in the input;

[0088] The second neural network is used to output a corresponding advantage estimation result based on the observed state in the input, and to update the first neural network by applying the advantage estimation result in a gradient ascent manner.

[0089] Understandably, the first neural network can be written as an actor model, which outputs actions based on the input state; the second neural network can be written as a critic model, which evaluates and scores the output actions of the actor network.

[0090] The agent collects experience through interaction with the environment and stores this experience for training. During training, a network with the same network structure as the agent and its corresponding critic network are trained and updated. The agent's network model interacting with the environment is updated every T steps. Using π... θ (a t |s t ) represents the policy of the PPO agent. A represents the old, unupdated strategy. t (s t ,a t The objective function (or loss function) is the advantage estimate of the critic network output, which is updated using gradient ascent in the actor network.

[0091]

[0092] Where θ represents the weights of the neural network, and ε = 0.1 is the hyperparameter of the cutoff function. The Critic network updates its weights using gradient descent, employing the following loss function:

[0093]

[0094] in, The target value, V, is calculated based on temporal difference learning (TD). μ (s t ) is the prediction value of the critic network.

[0095] To further improve the reliability and effectiveness of the forward prediction neural network and to further ensure the accuracy of the optimization of chiral metasurface structure parameters, in the chiral metasurface structure parameter optimization model training method provided in this application embodiment, the forward prediction neural network includes: a deep neural network for forward prediction;

[0096] The deep neural network is used to output a corresponding spectral response based on the input structural parameters, the spectral response including a circular dichroism (CD) spectrum.

[0097] To further improve the reliability and effectiveness of the feedforward predictive neural network and to further ensure the accuracy of chiral metasurface structure parameter optimization, a chiral metasurface structure parameter optimization model training method is provided in this application embodiment, see [link to relevant documentation]. Figure 4 The training method for optimizing the chiral metasurface structure parameters further includes the following content before step 100 or step 200:

[0098] Step 010: Obtain the structural parameters and corresponding spectral response labels of the chiral metasurface to obtain the corresponding training set.

[0099] Step 020: The training set is used to train the deep neural network for forward prediction so that the deep neural network outputs the spectral response corresponding to the structural parameters, and the objective function of the deep neural network is solved based on the spectral response and the spectral response label. The weights of the deep neural network are updated based on the Adam gradient descent method using the corresponding solution results.

[0100] To further improve the application efficiency, reliability, and effectiveness of the feedforward predictive neural network, and to further ensure the optimization accuracy of the chiral metasurface structure parameters and improve the optimization efficiency of the chiral metasurface structure parameters, the observation state data and the training set in the chiral metasurface structure parameter optimization model training method provided in this application embodiment further include: flag bits;

[0101] The flag bit is used to indicate whether the structural parameters of the uniquely corresponding chiral metasurface are left-handed or right-handed.

[0102] Specifically, feedforward predictive neural networks such as Figure 5 As shown, the network contains four hidden layers, each with 512 neurons, and the Leaky-ReLU activation function is used after each neuron. Notably, this network can simultaneously predict the CD spectrum of both left-handed and right-handed structures. Therefore, in addition to structural parameters, the neural network input includes a flag (I) to distinguish between left-handed and right-handed structures. The network output represents 500 discrete values ​​on the CD spectrum. The mean squared error loss function is used as the objective function for network training, as shown in Equation (4). The Adam gradient descent method is used to update the network weights.

[0103]

[0104] Where, N b CD represents the number of input data in a batch. pred and CD true These are the CD spectrum output by the neural network and the labels collected through FDTD, respectively.

[0105] To further improve the application efficiency, reliability, and effectiveness of feedforward predictive neural networks, and to further ensure the accuracy of chiral metasurface structure parameter optimization and improve the efficiency of chiral metasurface structure parameter optimization, a chiral metasurface structure parameter optimization model training method provided in this application embodiment includes the following steps in step 010:

[0106] Step 011: Collect the structural parameters and corresponding spectral response labels of the chiral metasurface based on the finite-difference time-domain algorithm.

[0107] This application also provides a PPO agent for performing all or part of the training method for optimizing the structure parameters of the chiral metasurface, see [link to relevant documentation]. Figure 6 The PPO agent specifically includes the following:

[0108] Action generation module 10 is used to optimize the proximal policy optimization (PPO) algorithm based on reinforcement learning algorithms, and to train the structure parameter optimization model of the chiral metasurface using the current observation state and reward, so that the structure parameter optimization model outputs the corresponding action. The observation state data includes the structure parameters of the chiral metasurface, and the action includes the change in the structure parameters.

[0109] Action sending module 20 is used to send the action to an environment that is in the same Markov decision system as itself, so that the forward prediction neural network in the environment outputs the spectral response corresponding to the action and generates a new reward corresponding to the action based on the spectral response.

[0110] The experience receiving module 30 is used to receive the next observation state and the new reward sent by the environment, so as to iteratively train the structure parameter optimization model again based on the next observation state and the new reward.

[0111] The embodiments of the intelligent agent provided in this application can be used to execute the processing flow of the embodiment of the chiral metasurface structure parameter optimization model training method in the above embodiments. Its functions will not be repeated here, but can be referred to the detailed description of the embodiment of the chiral metasurface structure parameter optimization model training method in the above embodiments.

[0112] As described above, the agent provided in this application intelligently optimizes the structural parameters of chiral metasurfaces based on reinforcement learning algorithms. By combining the proximal policy optimization algorithm and the forward prediction neural network in reinforcement learning, the reverse design process is modeled as a decision problem, satisfying a Markov decision process. Through interaction with the environment, the agent can learn an efficient optimization strategy. This strategy automatically searches for the optimal solution in the parameter space according to the optimization objective, thereby effectively ensuring the accuracy of the optimization of the structural parameters of the chiral metasurface. By incorporating the forward prediction network as part of the interactive environment, the response speed of the interactive environment can be greatly improved, thereby increasing the efficiency of the agent in collecting the data required for training, accelerating the training of the agent, and thus improving the accuracy and efficiency of the optimization of the structural parameters of the chiral metasurface. Furthermore, it can improve the reliability and effectiveness of the application of micro / nano devices based on chiral metasurfaces designed using the optimization results of the structural parameters output by the chiral metasurface structural parameter optimization model.

[0113] This application also provides a Markov decision system, as shown in Figure 2(a), which specifically includes the following:

[0114] Interactive PPO agents and environment;

[0115] The PPO agent is used to execute the aforementioned training method for the chiral metasurface structure parameter optimization model;

[0116] The environment is equipped with the feedforward prediction neural network, which outputs the spectral response corresponding to the action sent by the PPO agent, and generates a new reward corresponding to the action based on the spectral response.

[0117] This application also provides a micro / nano device based on a chiral metasurface, including a glass substrate and a double-layer chiral metal structure located on the surface of the glass substrate. The structural parameters of the double-layer chiral metal structure are set according to the optimization results corresponding to the structural parameters output by the structural parameter optimization model of the chiral metasurface.

[0118] The structural parameter optimization model of the chiral metasurface is pre-trained based on the aforementioned chiral metasurface structural parameter optimization model training method.

[0119] It is understood that the optimization result output by the chiral metasurface structural parameter optimization model refers to the final structural parameters output by the trained chiral metasurface structural parameter optimization model provided in the aforementioned embodiments after a series of changes.

[0120] To further illustrate this solution, this application also provides a specific application example of a training method for optimizing the structural parameters of chiral metasurfaces, which includes the following:

[0121] (I) Neural networks for forward prediction

[0122] This application utilizes reinforcement learning algorithms to learn efficient strategies for the optimal design of chiral metasurfaces. The reinforcement learning algorithm mainly consists of two parts: an agent and an environment, as shown in Figure 2(a). The environment has a significant impact on the learning effect and efficiency of the agent because it needs to receive the actions performed by the agent and provide feedback on the results. In our problem, the finite-difference time-domain (FDTD) algorithm can be used as part of the environment. However, verifying a structure using FDTD is often time-consuming, significantly reducing the learning efficiency of the agent. To improve the efficiency of training the agent, a deep neural network (DNN), i.e., a feedforward predictive neural network, is used instead of FDTD as part of the environment, as shown in Figure 2(b).

[0123] Forward predictive neural networks such as Figure 5As shown, the network contains four hidden layers, each with 512 neurons, and the Leaky-ReLU activation function is used after each neuron. Notably, this network can simultaneously predict the CD spectrum of both left-handed and right-handed structures. Therefore, in addition to structural parameters, the neural network input includes a flag (I) to distinguish between left-handed and right-handed structures. The network output represents 500 discrete values ​​on the CD spectrum. The mean squared error loss function is used as the objective function for network training, as shown in Equation (4). The Adam gradient descent method is used to update the network weights.

[0124]

[0125] Where, N b CD represents the number of input data in a batch. pred and CD true These are the CD spectrum output by the neural network and the labels collected through FDTD, respectively.

[0126] (II) Reinforcement Learning to Find Optimization Strategies

[0127] This application uses the Proximal Policy Optimization (PPO) algorithm in reinforcement learning to learn the optimization policy. The PPO reinforcement learning algorithm is developed based on TensorFlow and uses a feedforward predictive neural network as part of the environment. This network is developed based on PyTorch. The data and corresponding labels required to train this feedforward network are collected using the Finite-Difference Time-Domain (FDTD) algorithm. To increase the efficiency of data collection, the lumopt toolkit is used to interact with the FDTD algorithm to automatically generate and collect the dataset. The overall process is as follows: Figure 7 As shown, the details are as follows:

[0128] S1. Use the Python toolkit lumopt to interact with the finite-difference time-domain algorithm to automatically collect the parameters of left-handed and right-handed structures and their corresponding CD spectra.

[0129] S2. Add flags to the structural parameters of the two chiral structures to distinguish them, and mix the two data, keeping the corresponding spectral labels and structural parameters consistent.

[0130] S3, PyTorch builds a feedforward prediction neural network, and the network is trained using a collected and processed dataset.

[0131] S4 and Tensorflow are used to build the PPO algorithm, which uses a feedforward predictive neural network as part of the interactive environment. The agent interacts with the environment and learns the strategy of reverse design.

[0132] S5. The learned agent executes the strategy and interacts with the environment based on the feedforward prediction neural network to quickly find the optimal solution.

[0133] A reinforcement learning system consists of five parts: agent, state, reward, action, and environment. Reinforcement learning algorithms can be broadly categorized into policy-based and value-based algorithms based on policy type. This paper employs a currently effective proximal policy optimization algorithm to find a good optimization policy for a chiral hypersurface, and uses a feedforward predictive neural network to accelerate the training of this algorithm, such as... Figure 8 As shown, “Reward” represents a reward; “Episodes” represents a round.

[0134] (1) Agent: The agent is the core of the entire reinforcement learning system. It can perceive the state of the environment and, based on the reward signals provided by the environment, learn to select an appropriate action to maximize long-term reward value. The agent learns a series of mappings from environmental states to actions based on the rewards provided by the environment as feedback. The principle of action selection is to maximize the probability of future accumulated rewards. The selected action not only affects the reward at the current moment but also affects the reward at the next moment and even in the future. Therefore, the basic rule of the agent in the learning process is: if an action brings a positive reward from the environment, then this action will be strengthened, and vice versa. The method proposed in this patent uses the PPO algorithm. The agent is a neural network that outputs actions based on the input state. Here, it is named the actor network. In addition, there is a critic network used to evaluate and score the output actions of the actor network. The agent collects experience by interacting with the environment and stores the collected experience for training. During the training process, the network with the same network structure as the agent and its corresponding critic network are trained and updated. The network model of the agent interacting with the environment is updated every T steps. π is used. θ (a t |s t ) represents the policy of the PPO agent. A represents the old, unupdated strategy. t (s t ,a t The objective function (or loss function) is the advantage estimate of the critic network output, which is updated using gradient ascent in the actor network.

[0135]

[0136] Where θ represents the weights of the neural network, and ε = 0.1 is the hyperparameter of the cutoff function. The Critic network updates its weights using gradient descent, employing the following loss function:

[0137]

[0138] in, The target value, V, is calculated based on temporal difference learning (TD). μ (s t ) is the prediction value of the critic network.

[0139] (2) Environment: The environment receives a series of actions performed by the agent, evaluates the quality of these actions, and converts them into quantifiable rewards for the agent, without instructing the agent on how to learn the actions. The agent can only learn based on its historical experience. Simultaneously, the environment provides the agent with its current state information. Here, the current structural parameters are used as state information. After the agent performs corresponding actions on the environment, it generates new state information, i.e., new structural parameters. The feedforward prediction neural network, as part of the environment, outputs the corresponding CD spectrum based on the structural parameters, and calculates the reward based on the degree to which the CD spectrum satisfies the objective.

[0140] (3) Reward: The environment provides the agent with a quantifiable scalar feedback signal to evaluate the quality of the action taken by the agent at a certain time step. Reinforcement learning is based on the assumption of maximizing cumulative reward: In reinforcement learning, the goal of the agent in making a series of action choices is to maximize the future cumulative reward. The design of the reward is very important for the stability of the agent's learning. In this application, the reward is determined by the closeness of the output CD spectrum to the target, as shown in formula (7).

[0141]

[0142] Here, "target" refers to the target wavelength, meaning the larger the absolute value of the CD at the target wavelength, the better. We set three thresholds to highlight the importance of achieving different levels of the target, including T1 = 0.37, T2 = 0.4, and T3 = 0.5.

[0143] (4) State: The state is the environmental information of the agent, containing all the information the agent uses to make action selections. In the design of chiral metasurfaces, the input state of the agent consists of the structural parameters of the metasurface and the chiral flags that distinguish the input structure, denoted as s. t =(D t ,L t W t G t Q t ,I t ).

[0144] The process of reverse designing chiral metasurfaces by combining the PPO algorithm and DNN is as follows:

[0145] (a) Initialization phase: Initializing the environment and network parameters in the PPO algorithm.

[0146] (b) PPO agent observation state s t .

[0147] (c) The PPO agent executes the policy π(θ) old Based on the input state, it outputs corresponding actions, thereby changing the structural parameters and obtaining a new state.

[0148] (d) The environment will provide feedback based on the quality of this action, including reward, the next state, and whether to reset the environment's flag.

[0149] (e) Experience (s) t ,a t ,r t+1 ,s t+1 The data is stored in the experience pool, and a batch of data is collected from the experience pool to update the network parameters.

[0150] (f) Repeat be until the agent learns a good policy.

[0151] (III) Performance Analysis

[0152] Feedforward neural networks, as part of the environment in reinforcement learning, accelerate the training speed of agents and increase the flexibility and complexity of the tasks they can perform. During training, in addition to the structural parameters changing through actions, the chirality of the metasurface and the target wavelength also change randomly, with the target wavelength ranging from 985nm to 1250nm. This complex and changing environment allows the agent to acquire the ability to perform multiple tasks with a single training iteration. Figure 9 The diagram illustrates the reward variation during the PPO algorithm training process. As the number of training iterations increases, the reward gained by the agent through interaction with the environment grows larger and larger, eventually stabilizing. Due to the instability of data sampling during training, the reward obtained by the agent will fluctuate. The strategy that maximizes the reward can be saved as the optimal strategy.

[0153] The trained agent can interact with the environment to make rapid decisions, designing a chiral metasurface that meets the target through multi-step actions within 10 seconds. For example... Figures 10(a) to 10(d)Figure 11 shows the spectral response of the left-handed metasurface designed by the trained agent in interaction with the environment. Taking 1100 nm in Figure 10(a), 1035 nm in Figure 10(b), 1000 nm in Figure 10(c), and 930 nm in Figure 10(d) as targets, the agent can quickly search for the corresponding structural parameters to maximize the absolute value of the CD spectrum at the target wavelength. Figure 11 shows the spectrum of the right-handed metasurface designed by the agent, with the tested targets being 1100 nm in Figure 11(a), 1035 nm in Figure 11(b), 1000 nm in Figure 11(c), and 930 nm in Figure 11(d). In Figures 10(a) and 11(d), “Transmittance” represents transmittance; “Target” represents target wavelength; “Wavelength” represents wavelength; “Distance” represents the distance D between the upper and lower gold nanorods; “Length” represents the length L of the nanorod; “Width” represents the width W of the nanorod; “Gap” represents the gap G between the left and right gold nanorods; and “Period” represents the structural period.

[0154] The algorithm proposed in this application addresses the problem of obtaining the target spectral response by optimizing structural parameters in the design of chiral metasurfaces. Through reinforcement learning and interaction with the environment, it identifies the most efficient optimization strategy, enabling the acquisition of structural parameters for chiral metasurfaces that meet the optimization objectives in the shortest possible time. To improve the training speed and the complexity of the tasks learned by the reinforcement learning algorithm, a feedforward predictive neural network is used as part of the interactive environment, facilitating the exploration of its application in the reverse design of metasurfaces. Simulation results demonstrate that the proposed reinforcement learning algorithm can quickly learn an efficient optimization strategy, significantly reducing the optimization time and yielding satisfactory results. This algorithm can not only be applied to the intelligent design of chiral metasurfaces but can also be extended to the intelligent design of other micro / nano photonic devices.

[0155] This application also provides a computer device (i.e., an electronic device), which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the chiral metasurface structure parameter optimization model training method mentioned in the above embodiments. The processor and memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and memory via wired or wireless means.

[0156] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0157] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the chiral metasurface structure parameter optimization model training method in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby realizing the chiral metasurface structure parameter optimization model training method in the above method embodiments.

[0158] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0159] The one or more modules are stored in the memory, and when executed by the processor, the chiral metasurface structure parameter optimization model training method in the embodiment is executed.

[0160] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0161] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.

[0162] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.

[0163] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned chiral metasurface structure parameter optimization model training method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0164] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0165] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0166] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0167] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for training a chiral metasurface structure parameter optimization model, characterized in that, include: Based on the Proximal Policy Optimization (PPO) algorithm in reinforcement learning, a structural parameter optimization model for a chiral metasurface is trained using the current observation state and reward, so that the structural parameter optimization model outputs a corresponding action. The observation state data includes the structural parameters of the chiral metasurface, and the action includes the change in the structural parameters. The action is sent to an environment that is in the same Markov decision system as itself, so that the forward prediction neural network in the environment outputs the spectral response corresponding to the action, and a new reward corresponding to the action is generated based on the spectral response. The system receives the next observation state and the new reward sent by the environment, and then iteratively trains the structure parameter optimization model based on the next observation state and the new reward.

2. The method for training a chiral metasurface structure parameter optimization model according to claim 1, characterized in that, The structural parameter optimization model of the chiral metasurface includes: a first neural network and a second neural network; The first neural network is used to output the corresponding action based on the observed state in the input; The second neural network is used to output a corresponding advantage estimation result based on the observed state in the input, and to update the first neural network by applying the advantage estimation result in a gradient ascent manner.

3. The method for training a chiral metasurface structure parameter optimization model according to claim 1, characterized in that, The forward prediction neural network includes: a deep neural network for performing forward prediction; The deep neural network is used to output a corresponding spectral response based on the input structural parameters, the spectral response including a circular dichroism (CD) spectrum.

4. The method for training a chiral metasurface structure parameter optimization model according to claim 3, characterized in that, Before sending the action to an environment within the same Markov decision system, the method further includes: Obtain the structural parameters and corresponding spectral response labels of the chiral metasurface to obtain the corresponding training set; The training set is used to train a deep neural network for forward prediction, so that the deep neural network outputs the spectral response corresponding to the structural parameters. The objective function of the deep neural network is solved based on the spectral response and the spectral response label. The weights of the deep neural network are updated based on the Adam gradient descent method using the corresponding solution results.

5. The method for training a chiral metasurface structure parameter optimization model according to claim 4, characterized in that, The observed state data and the training set also include: flag bits; The flag bit is used to indicate whether the structural parameters of the uniquely corresponding chiral metasurface are left-handed or right-handed.

6. The method for training a chiral metasurface structure parameter optimization model according to claim 4, characterized in that, The acquisition of the structural parameters and corresponding spectral response labels of the chiral metasurface includes: The structural parameters and corresponding spectral response labels of chiral metasurfaces are collected based on the finite-difference time-domain algorithm.

7. A PPO intelligent agent, characterized in that, include: An action generation module is used to train a structural parameter optimization model for a chiral metasurface based on the proximal policy optimization (PPO) algorithm in reinforcement learning. The model is trained using the current observation state and reward, so that the structural parameter optimization model outputs a corresponding action. The observation state data includes the structural parameters of the chiral metasurface, and the action includes the change in the structural parameters. An action sending module is used to send the action to an environment that is in the same Markov decision system as itself, so that the forward prediction neural network in the environment outputs the spectral response corresponding to the action and generates a new reward corresponding to the action based on the spectral response. An experience receiving module is used to receive the next observation state and the new reward sent by the environment, so as to iteratively train the structure parameter optimization model again based on the next observation state and the new reward.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the chiral metasurface structure parameter optimization model training method as described in any one of claims 1 to 6.

9. A Markov decision system, characterized in that, include: Interactive PPO agents and environment; The PPO agent is used to execute the chiral metasurface structure parameter optimization model training method according to any one of claims 1 to 6; The environment is equipped with the feedforward prediction neural network, which outputs the spectral response corresponding to the action sent by the PPO agent, and generates a new reward corresponding to the action based on the spectral response.

10. A micro / nano device based on a chiral metasurface, comprising a glass substrate and a double-layer chiral metal structure located on the surface of the glass substrate, characterized in that, The structural parameters of the double-layer chiral metal structure are set using the optimization results corresponding to the structural parameters output by the structural parameter optimization model of the chiral metasurface. The structural parameter optimization model of the chiral metasurface is pre-trained based on the chiral metasurface structural parameter optimization model training method according to any one of claims 1 to 6.

Images