Metasurface optimization design method based on reinforcement learning

Abstract

The invention provides a metasurface optimization design method based on reinforcement learning, relates to the technical field of electromagnetic metasurfaces, and solves the problems that a metasurface design method based on supervised learning highly depends on a data set and the manufacturing cost of the data set is high. The method comprises the following steps: constructing a reinforcement learning environment model corresponding to a metasurface, and constructing a value network corresponding to a reinforcement learning algorithm; based on the constructed reinforcement learning environment model and the value network, through an intelligent agent and environment interaction mode, an objective function of the reinforcement learning environment model is updated, decision making of an intelligent agent is optimized, after a preset optimization requirement is met, the interaction process is stopped, and a metasurface optimization design result is output; wherein the learning environment model is reinforced to give an agent reward by comparing the difference between a target function value of a new state and a given optimization value; the method provided by the invention has stronger solving capability for nonlinear and uncertain complex environments, has the characteristics of intelligence and rapidness, and reduces data set dependency and manufacturing cost.

Description

Technical Field

[0001] The present invention relates to the technical field of electromagnetic supersurfaces, and in particular to a supersurface optimization design method based on reinforcement learning. Background Art

[0002] Electromagnetic metasurfaces, as a type of composite material with two-dimensional planar structure, have attracted widespread attention in the field of electromagnetics due to their unique beam steering capabilities. Compared with traditional three-dimensional metamaterials, metasurfaces have many advantages due to their planar characteristics, including but not limited to design flexibility, easy integration, and compatibility with curved structures. This technology has a wide range of applications, covering complex scenarios from radar cross-section reduction to beam deflection, abnormal reflection, abnormal refraction, and even multi-beam switching, demonstrating its great potential in modern communications, radar systems, optical devices, and stealth technology.

[0003] In 2014, Professor Cui Tiejun pioneered the concept of digital coding metasurfaces, marking a new era in metasurface design. Coding metasurfaces greatly simplify the design process by identifying different reflection phase units with binary codes. In this framework, the design challenge of metasurfaces is transformed into the problem of finding the optimal coding sequence and the corresponding unit structure. However, as the research deepens, the optimization method of metasurface design has also exposed some key problems.

[0004] At present, the optimization design of metasurfaces mainly relies on two categories of methods: traditional optimization algorithms and supervised learning-based methods. The former includes traversal algorithms, genetic algorithms, ant colony algorithms, etc. Although they perform well when dealing with low-dimensional problems, their computational complexity rises sharply when faced with large-scale, high-bit-coded metasurfaces, resulting in extended design cycles and low efficiency. Especially for high-bit-coded metasurfaces, the limitations of traditional algorithms are more significant because they often find it difficult to effectively explore complex solution spaces.

[0005] On the other hand, although supervised learning-based methods have shown great potential in data-driven optimization, their performance is highly dependent on high-quality, large-scale datasets. If the dataset is improperly constructed or insufficient, the trained model may not accurately predict the performance of the metasurface, thus affecting the accuracy and reliability of the design.

[0006] Given the limitations of existing metasurface design methods, it is urgent to develop a new optimization design strategy to overcome the challenges in high-bit encoding metasurface design. This strategy should be able to efficiently handle large-scale optimization problems while ensuring the accuracy and robustness of the design, thereby promoting electromagnetic metasurface technology to a higher level of application. Summary of the invention

[0007] Based on the current status of the background technology, the purpose of the present invention is to solve the problems of the hypersurface design method based on supervised learning, such as high dependence on the data set, high production cost of the data set and high-bit encoding of the hypersurface, and therefore proposes a hypersurface optimization design method based on reinforcement learning. The method of the present invention takes the hypersurface characteristics of interest as the optimization target, and through the interaction between the intelligent agent and the environment, grasps the operating laws of the environment, formulates corresponding strategies, and makes the hypersurface state parameters change towards the desired functions or technical indicators. The method of the present invention has a stronger ability to solve complex environments with nonlinearity and uncertainty, has the characteristics of intelligence and speed, and does not require the preparation of data sets in advance, reducing the dependence on data sets and production costs.

[0008] The present invention adopts the following technical solutions to achieve the purpose:

[0009] A hypersurface optimization design method based on reinforcement learning, the method comprising the following steps:

[0010] S1. Construct a reinforcement learning environment model corresponding to the hypersurface;

[0011] S2. Construct the value network corresponding to the reinforcement learning algorithm;

[0012] S3. Based on the constructed reinforcement learning environment model and value network, the objective function of the reinforcement learning environment model is updated through the interaction between the agent and the environment to optimize the decision-making of the agent;

[0013] S4. After the preset optimization requirements are met, the interaction process is stopped and the metasurface optimization design results are output.

[0014] Furthermore, in step S3, the intelligent agent is a whole composed of a value network, an action selection strategy and iterative updates, and the intelligent agent is a component of the reinforcement learning algorithm; the intelligent agent observes the environmental state in the reinforcement learning environment model, obtains the action value corresponding to the executable decision action through the value network, and makes a decision action according to the action selection strategy; the decision action is given to the reinforcement learning environment model, the environmental state is updated, and the corresponding objective function value is also updated. By comparing the gap between the objective function value of the new state and the given optimization value, the reinforcement learning environment model rewards the intelligent agent; the intelligent agent performs iterative updates based on the process of making a decision action and obtaining a reward each time.

[0015] Furthermore, in step S3, the reinforcement learning algorithm adopts the DQN algorithm. In the DQN algorithm, two independent value networks are used as the current value network Q(θ) and the target value network Q(θ - ), through the target value network Q(θ - ) keeps the target value of the reinforcement learning process segmented and stable. The current value network is responsible for observing the current state of the environment s t And give a decision action a t, the environment updates its state after receiving the action and calculates the new state s t+1 The corresponding objective function value, and then returns the reward r according to the reward rule t+1 ; The sample data (s t ,a t ,s t+1 ,r t+1 ) is stored in the experience replay pool. When the number of sample data in the experience replay pool exceeds the batch training number b, data is randomly and uniformly extracted from it for training, and the extracted batch data is input into the current value network and the target value network respectively. The error between the output values ​​of the two value networks is used as the loss function L t , the formula is as follows:

[0016]

[0017] In the formula, γ is the discount factor; the parameters of the current value network are updated through error back propagation, and the parameters of the current value network are assigned to the target value network every g time steps. After multiple iterations, the decision-making ability of the current value network is improved, and the intelligent agent can eventually find a state parameter design whose objective function value is less than the given optimization value.

[0018] The present invention also provides a computer device system, including a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the aforementioned supersurface optimization design method based on reinforcement learning.

[0019] The present invention also provides a computer-readable storage medium on which a computer program / instruction is stored, characterized in that: when the computer program / instruction is executed by a processor, the steps of the aforementioned supersurface optimization design method based on reinforcement learning are implemented.

[0020] In summary, due to the adoption of this technical solution, the beneficial effects of the present invention are as follows:

[0021] The reinforcement learning mechanism enables the intelligent agent to autonomously explore the unknown environment. By repeatedly making decisions, observing results and adjusting strategies, the intelligent agent can gradually understand the rules of the environment, thereby formulating effective strategies and guiding the evolution of the metasurface state parameters towards the desired functions and technical indicators.

[0022] Compared with traditional optimization algorithms such as particle swarm optimization, the method of the present invention is better at handling complex scenarios of nonlinearity and uncertainty, showing stronger solving ability and faster convergence speed, and is therefore particularly suitable for solving the design problems of high-bit coded metasurfaces.

[0023] Unlike supervised learning methods that require the support of a large number of high-quality data sets, the present invention achieves optimized design through direct interaction between the agent and the environment, avoiding high dependence on data sets and reducing the cost and workload of data preparation. Traditional design methods often rely on a large amount of manual intervention, which consumes time and manpower; while the present invention reduces manual participation, reduces time and manpower costs, and improves efficiency and economy.

[0024] Whether facing the challenges of large-scale module or high-bit coding metasurface design, the method provided by the present invention can demonstrate its superiority and provide new possibilities for the development of metasurface technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;

[0026] Figure 2 This is a schematic diagram of the median network structure of the present invention;

[0027] Figure 3 A schematic diagram of a model for interaction between an intelligent agent and an environment in the present invention;

[0028] Figure 4 It is a flowchart of the DQN algorithm in the present invention;

[0029] Figure 5 Schematic diagram of the metasurface phase distribution optimization results in the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0031] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] Example 1

[0033] A hypersurface optimization design method based on reinforcement learning, Figure 1 The overall process of the method is briefly described in FIG. 1 , and each step of the method is summarized as follows:

[0034] S1. Construct a reinforcement learning environment model corresponding to the hypersurface;

[0035] S2. Construct the value network corresponding to the reinforcement learning algorithm;

[0036] S3. Based on the constructed reinforcement learning environment model and value network, the objective function of the reinforcement learning environment model is updated through the interaction between the agent and the environment to optimize the decision-making of the agent;

[0037] S4. After the preset optimization requirements are met, the interaction process is stopped and the metasurface optimization design results are output.

[0038] This embodiment will introduce the details of each step in detail according to the above step sequence.

[0039] In step S1, the reinforcement learning environment model is a mathematical model or a simulation environment model, which is composed of the metasurface state information and physical laws, including state space, action space, state update and reward mechanism modules.

[0040] In step S1, the metasurface is composed of M×N array elements with a side length of D, where the phase corresponding to the (m,n)th array element is

[0041] In the reinforcement learning environment model, multiple parameters describing the state of the metasurface together constitute the state space of the metasurface. These parameters include phase distribution, array element period D, etc.; the state space contains all possible states in the environment. For example, the state space corresponding to the phase distribution is [0,2π].

[0042] The behavior of changing the state of the metasurface is recorded as a decision action, and multiple decision actions constitute the action space; the action space can be a continuous value or a discrete value.

[0043] When the reinforcement learning environment model receives the decision action made by the intelligent agent, it calculates the corresponding objective function value and realizes the state update; the objective function value represents the specific functions or technical indicators required by the hypersurface.

[0044] The reward mechanism is realized by comparing the gap between the objective function value in the new state after a decision action is taken and the given optimization value. The reinforcement learning environment model uses the reward as a feedback signal to the intelligent agent. Through this feedback signal, the effectiveness of the action taken by the intelligent agent at a certain moment can be evaluated, and the reward can promote the intelligent agent to move towards the desired result faster.

[0045] In step S2, the value network is a neural network that maps the environment state to the action value; the value network includes an input layer, an output layer and multiple hidden layers, and a corresponding activation function is used after each hidden layer. The activation function is used to increase the nonlinear fitting ability of the value network; the input of the value network is the environment state in the reinforcement learning environment model, and the output is the action value corresponding to all the decision actions that the intelligent agent can perform.

[0046] In step S3, the intelligent agent is a whole composed of a value network, an action selection strategy and iterative updates. At the same time, the intelligent agent is also an important component of the reinforcement learning algorithm; the intelligent agent observes the environmental state in the reinforcement learning environment model, obtains the action value corresponding to the executable decision action through the value network, and makes a decision action according to the action selection strategy; the decision action is given to the reinforcement learning environment model, the environmental state is updated, and the corresponding objective function value is also updated. By comparing the gap between the objective function value of the new state and the given optimization value, the reinforcement learning environment model rewards the intelligent agent; the intelligent agent performs iterative updates based on the process of making a decision action and obtaining a reward each time, thereby improving its decision-making ability.

[0047] In this embodiment, the reinforcement learning algorithm adopts the DQN algorithm. In the DQN algorithm, two independent value networks are used as the current value network and the target value network respectively. The target value network is used to keep the target value segmentation of the reinforcement learning process stable.

[0048] For the sample data obtained by each iteration of the agent, the experience replay pool is used to store the sample data, and the agent is trained again by randomly and evenly extracting the stored sample data from the experience replay pool. This method can make the samples participating in the training meet or approximately meet independent and identical distribution, eliminating the correlation between samples; at the same time, the experience replay process can also improve data utilization, because a sample can be used multiple times.

[0049] Specifically, two independent value networks are used as the current value network Q(θ) and the target value network Q(θ - ), through the target value network Q(θ - ) keeps the target value of the reinforcement learning process segmented and stable. The current value network is responsible for observing the current state of the environment s t And give a decision action a t , the environment updates its state after receiving the action and calculates the new state s t+1 The corresponding objective function value, and then returns the reward r according to the reward rule t+1 ; The sample data (s t ,a t ,s t+1 ,r t+1) is stored in the experience replay pool. When the number of sample data in the experience replay pool exceeds the batch training number b, data is randomly and uniformly extracted from it for training, and the extracted batch data is input into the current value network and the target value network respectively. The error between the output values ​​of the two value networks is used as the loss function L t , the formula is as follows:

[0050]

[0051] In the formula, γ is the discount factor; the parameters of the current value network are updated through error back propagation, and the parameters of the current value network are assigned to the target value network every g time steps. After multiple iterations, the decision-making ability of the current value network is improved, and the intelligent agent can eventually find a state parameter design whose objective function value is less than the given optimization value.

[0052] As a preferred embodiment of this embodiment, in step S3, the action selection strategy adopts the ε-greedy strategy, in which the value ε ranges from 0 to 1, representing the probability of the agent randomly selecting a decision action.

[0053] The advantage of using the ε-greedy strategy is that in the early stage of iterative training, the agent knows nothing about the environment, which makes it have a certain random exploration behavior, which can help it recognize the environment faster; therefore, in the iterative update process of the agent, the value ε of the ε-greedy strategy gradually decreases with the increase of the number of iterative training steps, which means that the agent's exploratory nature becomes smaller and smaller, and it is more inclined to use the information it has mastered, that is, when the agent does not randomly select the decision action through the ε-greedy strategy, it selects the action corresponding to the maximum value of the action value output by the value network as the decision action.

[0054] Finally, in step S4, the iteration is terminated when the number of iteration steps of the agent reaches the preset upper limit, the iteration is stopped manually, or the environmental state parameters of the reinforcement learning environment model that meet the given optimization value have been found. If the environmental state parameters whose corresponding objective function value is less than the given optimization value have been found, the state parameters are output as the hypersurface optimization design results; if they are still not found, the environmental state parameters closest to the given optimization value at the time of iteration termination are output as the hypersurface optimization design results.

[0055] The above method of this embodiment can be applied to a computer device system, which includes a memory, a processor and a computer program stored in the memory. When the processor executes the computer program, the steps of the aforementioned supersurface optimization design method based on reinforcement learning are implemented.

[0056] At the same time, the above method of this embodiment can be recorded in a computer-readable storage medium, which stores a computer program / instruction. When the computer program / instruction is executed by the processor, the steps of the aforementioned supersurface optimization design method based on reinforcement learning are implemented.

[0057] In this embodiment, these computer programs / instructions may be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more steps in the method.

[0058] In summary, the hypersurface optimization design method based on reinforcement learning of the present invention not only solves many problems existing in traditional methods, but also opens up a new path for hypersurface design, and has significant technical advantages and broad application prospects.

[0059] Example 2

[0060] On the basis of Example 1, this example introduces the application process of the hypersurface optimization design method based on reinforcement learning in Example 1 based on a specific example scenario.

[0061] This embodiment aims at a 1-bit programmable metasurface and realizes the radar cross section (RCS) reduction function by optimizing the phase distribution; the objective function is the RCS value formula. Given the RCS target value of -20dB, the optimization goal is to make the RCS value of the metasurface less than or equal to -20dB through phase arrangement, combined with Figure 1 Given the overall process, the application steps of this scenario are as follows:

[0062] Step 1: Construct a reinforcement learning environment model corresponding to the hypersurface

[0063] Construct a mathematical model corresponding to the metasurface, including state space, action space, state update and reward setting. The size of the metasurface array is M×N (M rows and N columns), and the phase distribution of the array element unit is used as the metasurface state. According to the array size, the state space size is M×N, and the unit phase value is 0 or π.

[0064] In this embodiment, the 1-bit metasurface is optimized and designed. The unit phase can only be 0 or π, so the decision action is defined as adding π to the original unit phase. For example, each action only changes the phase value of one element unit, so the action may be any discrete value from 0 to (M×N-1), and the action space is {0,...,(M×N-1)}.

[0065] The reward setting is related to the objective function value. In this embodiment, the objective function is the characteristic of the metasurface in terms of RCS reduction. When the RCS value corresponding to the phase distribution is less than or equal to the given target value, the agent can get a reward. The reward value is set to 50, and the reward is 0 at other times.

[0066] In this embodiment, the process of calculating the RCS value of the metasurface is as follows:

[0067] The metasurface is composed of M×N array elements with a side length of D, where the phase corresponding to the (m,n)th array element is is 0° or 180°. When a plane wave is incident on the metasurface, the far-field function of the metasurface is expressed as follows:

[0068]

[0069] In the formula, θ, are the pitch angle and azimuth angle corresponding to the direction of arbitrary wave vector, k is the vacuum wave number, k = 2π / λ, λ is the wavelength of the incident wave in vacuum, D is the array element period, is the unit far-field directivity coefficient of the corresponding array element (pattern function of a lattice).

[0070] Substituting the phase of the corresponding array element into the expression, the directivity function of the metasurface can be written as follows:

[0071]

[0072] According to the far-field direction function and directivity function, the RCS value calculation formula is as follows:

[0073]

[0074] Therefore, the reward setting rule is: reward the agent by comparing the difference between the RCS value corresponding to the new state after taking the action and the given RCS target value.

[0075] In this embodiment, the metasurface related parameter values ​​used are shown in Table 1 below.

[0076] Table 1 Metasurface related parameter values

[0077]

[0078] Step 2: Construct the value network corresponding to the reinforcement learning algorithm

[0079] The value network includes an input layer, an output layer and multiple hidden layers. This embodiment takes four fully connected layers as an example. The ReLU activation function is used after the first three fully connected layers. The input dimension of the value network is the state space dimension M×N, and the output dimension is the action space dimension M×N. The value network structure is as follows: Figure 2 shown.

[0080] Step 3: The agent interacts with the environment to optimize the objective function

[0081] In this embodiment, the interaction between the agent and the environment can be seen in Figure 3 As shown in the figure, the intelligent agent is a whole composed of the value network, action selection strategy and iterative update. At the same time, the intelligent agent is also an important part of the reinforcement learning algorithm; the intelligent agent observes the environmental state in the reinforcement learning environment model, obtains the action value corresponding to the decision action that can be performed through the value network, and makes a decision action according to the action selection strategy; the decision action is given to the reinforcement learning environment model, the environmental state is updated, and the corresponding objective function value is also updated. By comparing the gap between the objective function value of the new state and the given optimization value, the reinforcement learning environment model rewards the intelligent agent; the intelligent agent performs iterative updates based on the process of making decision actions and obtaining rewards each time, thereby improving its decision-making ability.

[0082] The DQN algorithm process used by the method can be found in Figure 4 As shown in the figure, two independent networks with consistent structures are introduced, namely the current value network and the target value network. At the beginning of training, the two networks adopt the same training parameters; during the training process, the current value network interacts with the environment to obtain samples and stores them in the experience replay pool. When storing samples, if the replay pool is full, the earliest stored samples are overwritten to ensure the real-time nature of the samples.

[0083] When the number of samples is greater than the number of batch training, batch data can be extracted from the playback pool and input into the current value network and the target value network to calculate the value function respectively. The value functions output by the two networks are compared to obtain the loss function, and the current value network parameters are updated based on the loss function. In this embodiment, the current value network parameters are copied to the target value network every time g time steps of training are completed.

[0084] In this process, the environment updates its state after receiving the action decision of the agent, calculates the far-field RCS value, and returns the reward and end flag. The DQN algorithm continuously iterates and learns according to this process and updates the strategy. The target value network replacement step length g is set to 100 steps, the discount factor γ is set to 0.99, and the experience replay pool capacity is 10,000.

[0085] Step 4: Terminate the interaction and output the results

[0086] After inputting the given RCS value and other constraints, as shown in Table 1, the output phase distribution result is as follows Figure 5As shown, black and white represent phases π and 0 respectively.

Claims

1. A hypersurface optimization design method based on reinforcement learning, characterized in that: The method comprises the following steps: S1. Construct a reinforcement learning environment model corresponding to the hypersurface; S2. Construct the value network corresponding to the reinforcement learning algorithm; S3. Based on the constructed reinforcement learning environment model and value network, the objective function of the reinforcement learning environment model is updated through the interaction between the agent and the environment to optimize the decision-making of the agent; S4. After the preset optimization requirements are met, the interaction process is stopped and the metasurface optimization design results are output.

2. The method for optimizing the supersurface based on reinforcement learning according to claim 1, characterized in that: In step S1, the reinforcement learning environment model is a mathematical model or a simulation environment model, which is composed of the metasurface state information and physical laws, including state space, action space, state update and reward mechanism modules.

3. The method for optimizing the design of a supersurface based on reinforcement learning according to claim 1, characterized in that: In step S1, the metasurface is composed of M×N array elements with a side length of D, where the phase corresponding to the (m,n)th array element is In the reinforcement learning environment model, multiple parameters describing the state of the hypersurface together constitute the state space of the hypersurface, the behavior of changing the state of the hypersurface is recorded as a decision action, and multiple decision actions constitute the action space; the update of the hypersurface state is driven by the decision actions made by the intelligent agent; the reward mechanism is realized by comparing the gap between the objective function value in the new state after a decision action is taken and the given optimization value, and the reinforcement learning environment model uses the reward as a feedback signal to the intelligent agent.

4. The method for optimizing the design of a supersurface based on reinforcement learning according to claim 1, characterized in that: In step S2, the value network is a neural network that maps the environment state to the action value; the value network includes an input layer, an output layer and multiple hidden layers, and a corresponding activation function is used after each hidden layer. The activation function is used to increase the nonlinear fitting ability of the value network; the input of the value network is the environment state in the reinforcement learning environment model, and the output is the action value corresponding to all the decision actions that the intelligent agent can perform.

5. The method for optimizing the design of a supersurface based on reinforcement learning according to claim 1, characterized in that: In step S3, the agent is a whole composed of the value network, the action selection strategy and the iterative update. The agent is a component of the reinforcement learning algorithm. The agent observes the environmental state in the reinforcement learning environment model, obtains the action value corresponding to the decision action that can be performed through the value network, and makes a decision action according to the action selection strategy. The decision action is given to the reinforcement learning environment model, the environmental state is updated, and the corresponding objective function value is also updated. By comparing the gap between the objective function value of the new state and the given optimization value, the reinforcement learning environment model gives the agent a reward. The agent is iteratively updated based on the process of making decisions and obtaining rewards each time.

6. The method for optimizing the design of a supersurface based on reinforcement learning according to claim 5, characterized in that: In step S3, the reinforcement learning algorithm adopts the DQN algorithm. In the DQN algorithm, two independent value networks are used as the current value network and the target value network respectively. The target value network is used to keep the target value segmentation of the reinforcement learning process stable; for the sample data obtained by each iterative update of the agent, the experience replay pool is used to store the sample data, and a new iterative training process of the agent is performed by randomly and uniformly extracting the stored sample data from the experience replay pool.

7. The method for optimizing the design of a supersurface based on reinforcement learning according to claim 5, characterized in that: In step S3, the action selection strategy adopts the ε-greedy strategy, in which the value ε ranges from 0 to 1, representing the probability of the agent randomly selecting a decision action; during the iterative update process of the agent, the value ε of the ε-greedy strategy gradually decreases with the increase of the number of iterative training steps; based on the application of this strategy, when the agent does not randomly select a decision action, the agent selects the action corresponding to the maximum value of the action value output by the value network as the decision action.

8. The method for optimizing the design of a supersurface based on reinforcement learning according to claim 1, characterized in that: In step S4, when the number of iteration steps of the intelligent agent reaches a preset upper limit, the iteration is stopped manually, or the environmental state parameters of the reinforcement learning environment model that meet the given optimization value have been found, the iteration is terminated; the environmental state parameters that are less than or closest to the given optimization value when the iteration terminates are output as the hypersurface optimization design results.

9. A computer device system, comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the supersurface optimization design method based on reinforcement learning as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the supersurface optimization design method based on reinforcement learning described in any one of claims 1 to 8 are implemented.

Images