Structural optimization method for free-form surface super-structure surface deflector based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based structure optimization method for a free-form surface meta-structure surface deflector, and relates to the technical field of optics. The method comprises the steps that a reinforcement learning framework is constructed, the optimization problem of the free form metasurface is reconstructed into an interaction system composed of an intelligent agent and an environment, electromagnetic simulation is conducted on the environment based on strict coupling wave analysis (RCWA), the effects of the actions are evaluated, and deflection efficiency obtained through simulation is converted into rewards; the intelligent agent follows a strategy, a state-action value function is constructed by selecting actions, and behaviors of the intelligent agent are guided to maximize long-term accumulated rewards; an experience playback buffer area is used for storing a state transition tuple, exploration and utilization are carried out through an algorithm, parameters of a parameterized network are copied regularly, the network is updated and exploration and utilization are balanced by minimizing Huber loss, and it is ensured that a path reaching an optimal solution can be found and learned; and the problem of optimization of the high-complexity one-dimensional free-form metasurface is solved.

Description

Technical Field

[0001] The present invention belongs to the field of optical technology, and in particular relates to a free-form surface metasurface deflector structure optimization method based on reinforcement learning. Background Art

[0002] Metasurfaces, optical devices with subwavelength structures that can control the properties of light with unprecedented spatial resolution, have attracted great attention in the past decade. With their ability to manipulate the phase, amplitude, and polarization of light, metasurfaces have been used in a variety of applications, ranging from achromatic beam focusing, vortex beam generation, and holography to optical computing and quantum optical information processing.

[0003] The exploding demand for high-performance metasurfaces has prompted researchers to seek efficient and effective methods for inverse design, a method for finding device structures that exhibit the desired optical response. Traditional inverse design methods include genetic algorithms, particle swarm methods, and adjoint-based algorithms. While these methods exhibit comparable performance, their design space is limited to a small number of structures. With the advent of deep learning, which uses artificial neural networks to approximate arbitrary nonlinear functions, more research has begun to tackle high-dimensional problems. However, early deep learning-based inverse design methods restricted the geometry of constituent device elements to fixed shapes, such as circles or rectangles, and only varied the size or position of the shapes to limit the complexity of the design space to a computationally tractable amount.

[0004] For example, Chinese patent CN114755846A discloses a reverse-design photonic simulator based on phase change materials and a reverse-design method, comprising: a chip, comprising a silicon substrate layer, a silicon oxide layer, a planar optical waveguide and a phase change material layer arranged in sequence from bottom to top; the phase change material layer does not cover the input port and the output port; an optical control module, arranged above the chip, capable of three-dimensional translation, for performing laser irradiation on different positions in the phase change material layer to change the state of the phase change material at the corresponding position; a signal light source, arranged on one side of the input port of the planar optical waveguide, for inputting a laser signal; and a photoelectric detection module, arranged on one side of the output port of the planar optical waveguide, for detecting the optical power of the output optical signal to obtain the optical response of the chip; preferably, the output port of the planar optical waveguide is also integrated with a nonlinear material.

[0005] Free-form design, an area of ​​structural optimization where the shape is not well-defined, has been challenging due to the large dimensionality of its search space until recent work has proposed solutions for free-form optimization based on generative adversarial networks (GANs) and adjoint-based generative networks. However, these optimization methods either require a priori datasets with sufficiently high network training performance or suffer from large variations in the performance of the generated devices. Many solutions have been proposed, including recent work on deep Bayesian optimization, but the huge design space of free-form device design has not yet been overcome.

[0006] Therefore, the present invention provides a free-form surface metasurface deflector structure optimization method based on deep reinforcement learning. Summary of the invention

[0007] The purpose of the present invention is to provide a free-form metasurface deflector structure optimization method based on reinforcement learning. The process of the deep reinforcement learning deflector structure optimization method includes reinforcement learning RL framework construction, reward function definition, state-action value function definition, and experience replay network, which solves the high-complexity one-dimensional free-form metasurface optimization problem.

[0008] In order to solve the above technical problems, the present invention is achieved through the following technical solutions: The present invention is a free-form surface metasurface deflector structure optimization method based on reinforcement learning, comprising the following steps: Step S1, reformulating the free-form optimization of the meta-surface into a reinforcement learning (RL) framework, wherein the framework defines two basic concepts: an agent and an environment; Step S2: The environment performs electromagnetic simulation based on rigorous coupled wave analysis (RCWA) to calculate the deflection efficiency of a given super-polymer structure. , the deflection efficiency Convert to rewards ; Step S3: Define a strategy , choose an action under the action of the strategy Constructing state-action value function ; Step S4: Use the experience replay buffer to store the training data generated during the training process, where the training data includes state, action, reward, and next state; The algorithm periodically replicates the parameterized network parameters to minimize the Huber loss.

[0009] Furthermore, in step S1, the metasurface structure information is represented as a 1D array, wherein each element represents a unit, and the corresponding value is +1 or -1, indicating that the corresponding unit is filled with silicon Si or air, respectively.

[0010] Furthermore, in step S3: The Strategy represents the probability distribution S of action a in a given state, and selects an action at step t , using a neural network to approximate the state-action value function ; For any finite Markov decision process (MDP), taking actions according to the optimal policy for each state leads to an approximately optimal state-action value function Q* based on the Bellman optimality equation. The optimal policy is implicitly defined as and The argmax operator The goal is to find a nearly optimal state-action value function Q*.

[0011] Furthermore, the neural network that approximates the optimal state-action value function Q* is a neural network architecture of an intelligent agent consisting of two fully connected hidden layers; the input is a 1-dimensional array representing the device state; the dimension of each hidden layer of the neural network is twice the dimension of the input; the output is the predicted state-action value for each action ; Find the parameters through the optimization process of temporal difference learning The optimal set of .

[0012] Furthermore, it also includes: a finite Markov decision process MDP, wherein the finite Markov decision process MDP includes: In a neural network with an approximate optimal state-action value function Q*, the agent takes an action given the state of the environment and receives a new state and reward; Generate Sequence , from the initial state To the final state The entire sequence is labeled as a trajectory or episode, where the state is , the action is , the reward is .

[0013] Furthermore, an internal search algorithm is designed in the finite Markov decision process MDP to enable the agent to explore new experiences from unpredictable actions and generate optimal actions; the discounted sum of future rewards is maximized and the return of the event is calculated. ; For any given time t, where is a discount factor that assigns less weight to rewards received from further into the future.

[0014] Furthermore, in step S4, an additional network is introduced , periodically from The parameters are copied in order to stabilize possible oscillations or divergences; the neural network model will It is parameterized as a fully connected neural network with two hidden layers (2N=128 nodes) and rectified linear unit (ReLU) activation function.

[0015] Furthermore, the neural network approximating the optimal state-action value function Q* The training process is divided into the data collection phase and the network update phase: In the data collection phase, the agent goes through many episodes, each episode starts from the initial state Start, based on Algorithm, for a given state , with a certain probability , randomly select an action Exploration is performed; otherwise, the agent chooses the action that results in the maximum predicted state-action value; electromagnetic simulation is performed at each state transition to evaluate the reward , .

[0016] Furthermore, in step S4, the method for calculating the Huber loss is: Sample a small batch (preset batch) of transformation data from the experience replay buffer and use it to calculate the Huber loss; Huber loss: ; in, Use additional network Calculated Bellman objective; The parameterized network is updated via stochastic gradient descent using the Adam optimizer.

[0017] The present invention has the following beneficial effects: The present invention defines two key concepts in the reinforcement learning RL network: the agent and the environment; the agent takes actions given the state of the environment and receives new states and rewards, and the environment performs electromagnetic simulation based on rigorous coupled wave analysis (RCWA) to calculate the deflection efficiency of a given hyper-polymer structure. , and then convert it into rewards ; By defining a strategy , choose an action under the action of this strategy Constructing state-action value function ; Use the experience replay buffer to store the training data generated during the training process, including state, action, reward, and next state; use The algorithm periodically replicates the parameterized network parameters to minimize the Huber loss.

[0018] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0020] Figure 1 This is a flowchart for implementing reinforcement learning (RL) in the present invention; Figure 2 A neural network architecture diagram of the approximate optimal state-action value function Q* of the present invention; Figure 3 Additional network for the present invention Training flow chart. DETAILED DESCRIPTION

[0021] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0022] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0023] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0024] As used in the specification of this application and the appended claims, the term "if" may be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [the described condition or event] is detected" may be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [the described condition or event] is detected" or "in response to detecting [the described condition or event]" depending on the context. In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0025] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0026] See also Figure 1-3 As shown, the present invention is a free-form surface meta-structure deflector structure optimization method based on reinforcement learning. The process of the deep reinforcement learning deflector structure optimization method includes reinforcement learning RL framework construction, reward function definition, state-action value function definition, and experience replay network. The reinforcement learning RL framework is used to define two key concepts: agent and environment. The agent changes its state s by taking action a on the environment, obtains reward r, and generates a sequence , thereby learning to select the optimal action; using RCWA electromagnetic simulation to calculate the +1-order diffraction efficiency as a reward, the higher the efficiency, the greater the reward; the state-action value function approximates the discounted sum Q* function of future rewards through a neural network architecture; the experience replay network model is divided into two branches: data collection and network update. At each state transition, a data structure consisting of Then, in the network update phase, a small batch of transformations is sampled from the buffer to update the network to find the optimal state-action value function Q*, which includes: S1. The free-form optimization of the meta-surface is reformulated as a reinforcement learning (RL) framework, which defines two basic concepts: the agent and the environment. S2, Environment performs electromagnetic simulations based on rigorous coupled wave analysis (RCWA) to calculate the deflection efficiency of a given super-polymer structure , and then convert it into rewards ; S3. Define a strategy , choose an action under the action of this strategy Constructing state-action value function ; S4. Use the experience replay buffer to store the training data generated during the training process, including states, actions, rewards, and next states. The algorithm periodically replicates the parameterized network parameters to minimize the Huber loss.

[0027] As an embodiment provided by the present invention, preferably, the environment of the present invention is defined as a There are a possible structural space. Specifically, the space is divided into N units by the periodicity of the metasurface. Each unit can be silicon Si or air, and its value is +1 or -1.

[0028] As an embodiment provided by the present invention, preferably, the reinforcement learning RL network in step S1 describes the state of the environment at a given time t is a 1×N array. Each of its elements contains +1 or -1, representing silicon battery and gas battery respectively. The initial state is fixed to an array with only +1 values ​​to stabilize the initial learning process. An action corresponds to selecting a specific cell among N cells and switching its material between silicon and air, which is called "flipping" the cell. The agent's action changes the state of the environment to the subsequent state .

[0029] As an embodiment provided by the present invention, preferably, Figure 1 As shown, it is a flowchart of the implementation of reinforcement learning RL according to an embodiment of the present invention; the state State is represented by an array of all +1 , which represents the state of the environment at a certain time t. Action Action is defined as "flipping" a cell. The agent can perform an operation to change the state of a cell from +1 to -1, or from -1 back to +1, thereby changing the state of the environment. Reward Reward represents the feedback signal obtained by the agent from the environment after flipping the cell, which is used to guide the agent to learn which actions are beneficial. The agent selects an action based on the current state and obtains rewards from the environment to learn the optimal strategy. Environment Environment In this embodiment, the first-order diffraction is used to make the normally incident TM polarized plane wave at an angle Deflected one-dimensional Si eutectic structure. The Si eutectic sits on a SiO2 substrate with a thickness of 325 nm. One grating period is divided into N = 64 cells and represented as a 1 × N array, whose elements are +1 or -1, indicating whether a particular cell is filled with Si or air.

[0030] As an embodiment provided by the present invention, preferably, in the neural network of the approximate optimal state-action value function Q* in step S2, the agent takes an action of a given environmental state, and receives a new state and reward, generating a sequence , where the status is , the action is , the reward is , to describe the initial state To the final state The entire sequence is called a trajectory or episode, and the process is called a finite Markov decision process (MDP).

[0031] As an embodiment provided by the present invention, preferably, the agent takes an action given the environment state, and receives a new state and reward, generating a sequence , where the status is , the action is , the reward is , to describe the initial state To the final state The entire sequence is called a trajectory or episode. This process is called a finite Markov decision process (MDP), and the scalar reward function described in this embodiment is defined as:

[0032] Among them, the status is , the action is , the reward is , the sets of state S, action a, and reward R all have a finite number of elements.

[0033] Design an internal search algorithm to enable the agent to explore new experiences from unpredictable actions and generate actions that it considers to be optimal. Maximize the discounted sum of future rewards and calculate the return of the event , defined as: ; For any given time t, , where , is a discount factor that assigns less weight to rewards received from farther into the future.

[0034] As an embodiment provided by the present invention, preferably, an internal search algorithm is designed in a finite Markov decision process MDP, so that the agent can explore new experiences from unpredictable actions and generate actions that it considers to be optimal; the discounted sum of future rewards is maximized and the reward of the event is calculated. . For any given time t, where is a discount factor that assigns less weight to rewards received from further into the future.

[0035] As an embodiment provided by the present invention, preferably, the strategy described in step S3 represents the probability distribution S of action a in a given state, and selects an action at step t , using a neural network to approximate the state-action value function For any MDP, the Bellman optimality equation states that taking actions according to the optimal policy for each state leads to the optimal Q*, where the optimal policy is implicitly defined as the given state-action value and The argmax operator The goal is to find the optimal state-action value function Q*.

[0036] ; in Indicates the current environment status. Indicates action, Indicates execution of an action In Status The expected return.

[0037] For any MDP, the Bellman optimality equation states that taking actions according to the optimal policy for each state leads to the optimal Q*, where the optimal policy is implicitly defined as and The argmax operator The goal is to find the optimal state-action value function Q*:

[0038] like Figure 2, which is a neural network architecture diagram of the approximate optimal state-action value function Q* according to an embodiment of the present invention. The neural network is a neural network architecture of an intelligent agent consisting of two fully connected hidden layers. The input is a 1-dimensional array representing the device state. The dimension of each hidden layer of the neural network is twice the dimension of the input. Output is the predicted state-action value for each action .

[0039] The parameters are found through the optimization process of temporal difference learning The optimal set of . Introducing an additional network , periodically from The neural network model described in this embodiment will It is parameterized as a fully connected neural network with two hidden layers (2N=128 nodes) and a rectified linear unit (ReLU) activation function. The process can be expressed as the following updated equation:

[0040] in, is the learning rate.

[0041] As an embodiment provided by the present invention, preferably, the neural network that approximates the optimal state-action value function Q* is a neural network architecture of an intelligent agent consisting of two fully connected hidden layers. The input is a 1-dimensional array representing the device state. The dimension of each hidden layer of the neural network is twice the dimension of the input. The output is the predicted state-action value for each action The parameters are found through an optimization process of temporal difference learning The optimal set of .

[0042] As an embodiment provided by the present invention, preferably, an additional network is introduced , periodically from The neural network model described in this embodiment will It is parameterized as a fully connected neural network with two hidden layers (2N=128 nodes) and rectified linear unit (ReLU) activation function.

[0043] As an embodiment provided by the present invention, preferably, the neural network of the optimal state-action value function Q* in step S4 The training process can be roughly divided into two independent stages: data collection and network update. In the data collection stage, the agent goes through many episodes. Each episode starts from the initial state Start. This embodiment Algorithm, for a given state , with a certain probability , randomly select an action exploration; otherwise, the agent chooses the action that results in the maximum predicted state-action value. Electromagnetic simulation is performed at each state transition to evaluate the reward ; This embodiment describes Algorithm, for a given state , with a certain probability , randomly select an action exploration is performed; otherwise, the agent chooses the action that results in the maximum predicted state-action value. It starts at 0.9, decays linearly to 0.01, and remains constant for enough learning steps so that the agent can make better decisions using its experience after enough exploration. Electromagnetic simulations are performed at each state transition to evaluate the reward .

[0044] As an embodiment provided by the present invention, preferably, in the network update phase, a small batch of conversion data is sampled from the experience replay buffer and used to calculate the Huber loss:

[0045] in Use additional network The Bellman objective is calculated. Then, the parameterized network is updated by stochastic gradient descent using the Adam optimizer.

[0046] The reward is positively correlated with the efficiency of the generated equipment. Each transition data is represented by Composed, stored in a fixed size of 10 -6 Experience replay buffer. When the total number of steps reaches twice the number of units in the structure (2N=128), an update ends to ensure that the agent completes the optimization within a limited and acceptable number of steps. During the network update phase, a small batch of transformation data is sampled from the experience replay buffer and used to calculate the Huber loss:

[0047] in, The network update is done by minimizing the Huber loss:

[0048] in, Use additional network The Bellman objective is calculated. Then, the parameterized network is updated by stochastic gradient descent using the Adam optimizer.

[0049] like Figure 3 As shown, the embodiment of the present invention is described Training flow chart. The information flow represented by arrows during the learning process of data collection (left) and network update process (right) of length M steps. Parameterized network Interact with the environment by exchanging states and actions. At each state transition, a buffer consisting of Then, during the network update phase, a mini-batch of transformations is sampled from the buffer to update the network.

[0050] In order to verify the effectiveness of the present invention, the method of the present invention was trained under 9 target conditions, where the wavelengths were 900nm, 1000nm and 1100nm, and the deflection angles were 50°, 60° and 70°, and compared with the greedy algorithm based on N=64, the adjoint-based method and GLOnet based on N=256.

[0051] The experimental simulation contents of the free-form surface metasurface deflection and its structural optimization method based on deep reinforcement learning of the present invention are as follows: During the deep reinforcement learning model training, the number of cells in a grating period is 64, the diffraction order of the rigorous coupled wave analysis RCWA algorithm is set to 40, and the playback buffer size is 10 6 , the discount factor value is 0.99; the total number of learning steps is set to 2,000,000; In the algorithm The value decay step period is set to 10 6 , with a minimum value of 0.01; 128 steps per training cycle, at least 5000 steps of experience before the start of neural network training, a learning rate of 0.001, a training batch size of 512, and one training run each time the train function is called.

[0052] The optimal structure found at the end of the whole process has a deflection efficiency of 98.4%, which is 7.4% higher than the previous state-of-the-art optimization result. The number of structures considered in the optimization process of the present invention is 2×10 -6 , which is the search space (10 17 ) a very small part (10 −11 ).

[0053] Tables 1-3 show the comparison results of three mainstream methods in the comparative experiments of the embodiments of the present invention, among which the variance of the RL method described in the present invention is much smaller than that of the adjoint-based method and GLOnet. Both the adjoint-based method and GLOnet are generative models based on device distribution, and their inherent result is a relatively large variance (10~20% p standard deviation).

[0054] On the other hand, the RL method described in the present invention finds devices whose efficiency has small variance (standard deviation < 2%p) across multiple random network initializations. This shows the robustness of the present invention in finding structures close to the optimal efficiency. It is found that the RL agent is able to find the most efficient device structure in 7 out of 9 test settings; the performance of the RL method also remains competitive in the other two cases. Table 1

[0055] Table 2

[0056] Table 3

[0057] A reinforcement learning-based free-form metasurface deflector structure optimization method is proposed. Two key concepts are defined in the reinforcement learning RL network: the agent and the environment. The agent takes actions given the state of the environment and receives new states and rewards. The environment performs electromagnetic simulation based on rigorous coupled wave analysis (RCWA) to calculate the deflection efficiency of a given meta-polymer structure. , and then convert it into rewards ; By defining a strategy , choose an action under the action of this strategy Constructing state-action value function ; Use the experience replay buffer to store the training data generated during the training process, including state, action, reward, and next state. The algorithm periodically replicates the parameterized network parameters to minimize the Huber loss.

[0058] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0059] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A free-form metasurface deflector structure optimization method based on reinforcement learning, characterized in that: The following steps are involved: Step S1, reformulate the free-form optimization of the meta-surface into a reinforcement learning (RL) framework and define the concepts of agent and environment; Step S2: The environment performs electromagnetic simulation based on rigorous coupled wave analysis to calculate the deflection efficiency of a given super-polymer structure. , the deflection efficiency Convert to rewards ; Step S3: Define a strategy , choose an action under the action of the strategy Constructing state-action value function ; Step S4, using the experience replay buffer to store the training data generated during the training process; use The algorithm periodically replicates the parameterized network parameters to minimize the Huber loss.

2. The free-form metasurface deflector structure optimization method based on reinforcement learning according to claim 1, characterized in that: In step S1, the environment is defined as a N The structure space is divided into N units by the periodicity of the metasurface, each unit is silicon Si or air, and the metasurface structure information is represented as a 1D array, where each element of the 1D array represents a unit, and the value corresponding to each unit is +1 or -1, indicating that the corresponding unit is filled with silicon Si or air, respectively.

3. The free-form metasurface deflector structure optimization method based on reinforcement learning according to claim 1, characterized in that: In step S3: The Strategy represents the probability distribution S of action a under a given state; Choose an action at step t , the approximate state-action value function of the neural network is ; For any finite Markov decision process MDP, based on the Bellman optimality equation, the optimal policy is implicitly defined as given the state-action value and The argmax operator , find the approximately optimal state-action value function Q*.

4. The free-form metasurface deflector structure optimization method based on reinforcement learning according to claim 3 is characterized in that: The neural network that approximates the optimal state-action value function Q* is a neural network architecture of an intelligent agent consisting of two fully connected hidden layers; the input is a 1-dimensional array representing the device state; the dimension of each hidden layer of the neural network is twice the dimension of the input; the output is the predicted state-action value for each action ; Find the parameters through the optimization process of temporal difference learning The optimal set of .

5. The free-form metasurface deflector structure optimization method based on reinforcement learning according to claim 4, characterized in that: The finite Markov decision process MDP includes: In a neural network with an approximate optimal state-action value function Q*, the agent takes an action given the state of the environment and receives a new state and reward; Generate Sequence , from the initial state To the final state The entire sequence is labeled as a trajectory or episode, where the state is , the action is , the reward is .

6. The method for free-form metasurface deflector structure optimization based on reinforcement learning according to claim 5, characterized in that: Design an internal search algorithm in the finite Markov decision process MDP and generate the optimal action; maximize the discounted sum of future rewards and calculate the return of the event ; For any given time t, where is a discount factor that assigns less weight to rewards received from further into the future.

7. The free-form metasurface deflector structure optimization method based on reinforcement learning according to claim 1, characterized in that: In step S4, an additional network is introduced , periodically from The neural network model will copy the parameters in It is parameterized as a fully connected neural network with two hidden layers and rectified linear unit activation function.

8. The method for free-form metasurface deflector structure optimization based on reinforcement learning according to claim 5, characterized in that: The neural network that approximates the optimal state-action value function Q* The training process is divided into the data collection phase and the network update phase: In the data collection phase, each episode experienced by the agent starts from the initial state Start, based on Algorithm, for a given state , with a certain probability , randomly select an action otherwise, the agent chooses the action that results in the maximum predicted state-action value. Electromagnetic simulations are performed at each state transition to evaluate the reward, , .

9. The method for free-form metasurface deflector structure optimization based on reinforcement learning according to claim 7, characterized in that: In step S4, the method for calculating the Huber loss is: Sample transformation data from the experience replay buffer and use it to calculate the Huber loss; Huber loss: ; in, Use additional network Calculated Bellman objective; The parameterized network is updated via stochastic gradient descent using the Adam optimizer.

10. The free-form metasurface deflector structure optimization method based on reinforcement learning according to claim 1, characterized in that: In step S4, the training data includes state, action, reward and next state.

Images