Design variable optimization method and system fusing reinforcement learning and cma-es

Abstract

The application discloses a design variable optimization method and system fusing reinforcement learning and CMA-ES, and relates to the field of engineering optimization design. The method comprises the following steps: determining truss body design variables; constructing a Gaussian process proxy model, a policy network and an evaluation network; iteratively updating a population and state information by using a CMA-ES algorithm until a set number of iterations is reached, and then outputting state information sequences under the current optimization trajectory; training the policy network and the evaluation network by using reinforcement learning based on the state information sequences under the current optimization trajectory, a state value function and an immediate reward; determining whether the current optimization trajectory reaches a set number of optimization trajectory training times; if yes, iteratively updating the population and the state information by using the CMA-ES algorithm based on the trained policy network of the reinforcement learning until a set number of iterations is reached, and then generating an optimal population. The application improves the optimization efficiency of the truss body design variables.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of engineering optimization design, in particular to a design variable optimization method and system fusing reinforcement learning and CMA-ES. BACKGROUND

[0002] In the field of engineering optimization design, the target object of engineering design is often relatively complex, and the optimization process is also a process of solving a high-dimensional optimization problem. Taking the lightweight design of a truss body structure as an example, since the truss body structure is relatively complex, there are many combinations of variable parameters such as the length, width, thickness and material of the key parts, which also leads to more than 100 dimensions of the truss body design variables.

[0003] In solving high-dimensional optimization problems, traditional heuristic algorithms need to construct a large solution space, and in the large solution space, there are a large number of invalid variable parameter combinations. Since the heuristic algorithm needs to perform iterative operations on each variable parameter combination, a large number of invalid variable parameter combinations will lead to a low adaptability of the heuristic algorithm, thereby affecting the optimization efficiency of the variable parameters.

[0004] Therefore, in the existing lightweight design of the truss body structure, the traditional heuristic algorithm is obviously not applicable, and it cannot effectively optimize the truss body design variable parameters. Even if a few groups of optimized truss body design variable parameters can be obtained by using the heuristic algorithm, the optimization efficiency is still low. SUMMARY

[0005] The application aims to provide a design variable optimization method and system fusing reinforcement learning and CMA-ES, which improves the optimization efficiency of the truss body design variables.

[0006] To achieve the above-mentioned purpose, the application provides the following solutions.

[0007] In a first aspect, the application provides a design variable optimization method fusing reinforcement learning and CMA-ES, which comprises the following steps:

[0008] determining a truss body design variable, taking a group of parameters of the truss body design variable as an individual, and taking a set of multiple groups of parameters of the truss body optimization variable as a population; the truss body design variable at least includes the side length, thickness and material of the horizontal and vertical beam;

[0009] construct a Gaussian process surrogate model, a policy network and an evaluation network; the Gaussian process surrogate model is used to determine the corresponding truss body mass according to each individual; the policy network is used to generate the corresponding evolutionary path change according to the state information of each population; the state information includes: population mean, population standard deviation and eigenvalue of population covariance matrix;

[0010] The population and state information are iteratively updated by using the CMA-ES algorithm until a set number of iterations is reached, and the state information sequence under the current optimization trajectory is output; the evolutionary path update formula in the CMA-ES algorithm adds the policy network; the determination process of the elite individual in the CMA-ES algorithm applies the Gaussian process surrogate model; the state information sequence includes state information under different iteration numbers;

[0011] The state information sequence under the current optimization trajectory is input into the evaluation network to obtain the state value function under the current optimization trajectory;

[0012] Based on the state information sequence under the current optimization trajectory, the state value function and the immediate reward, the policy network and the evaluation network are trained by using reinforcement learning;

[0013] Determine whether the current optimization trajectory reaches a set number of optimization trajectory training; if yes, obtain the trained policy network by reinforcement learning; if no, return to the step of iteratively updating the population and state information by using the CMA-ES optimization algorithm until a maximum number of iterations is reached, and output the state information sequence under the current optimization trajectory;

[0014] Based on the trained policy network by reinforcement learning, iteratively update the population and state information by using the CMA-ES algorithm until the optimal population is generated after a set number of iterations; the optimal population contains multiple optimal individuals; the optimal individual represents a set of optimal parameters of the truss body design variables.

[0015] Optionally, the evolutionary path update formula is:

[0016]

[0017] In the formula, is the evolutionary path of the tth iteration, is the evolutionary path of the (t+1) th iteration, α σ is the weight occupied by the historical evolutionary path, action is the evolutionary path change amount corresponding to the state information of the tth iteration of the population output by the policy network, μ (t) is the population mean of the tth iteration, μ (t+1) is the population mean of the (t+1) th iteration, σ (t) is the population standard deviation of the tth iteration.

[0018] Optionally, the construction process of the Gaussian process surrogate model specifically comprises:

[0019] establishing a parameter model according to the truss body design variables based on the geometric structure of the truss body;

[0020] performing finite element simulation on the parameter model to generate a structure performance dataset with the minimum truss body mass as the target; the structure performance dataset comprises multiple sets of parameters of the truss body design variables and the truss body mass corresponding to each set of parameters;

[0021] fitting the structure performance dataset by Gaussian process to obtain a Gaussian process surrogate model with one set of parameters of the truss body design variables as input and the truss body mass as output.

[0022] Optionally, both the policy network and the evaluation network are provided with two hidden layers, and the number of nodes in each hidden layer is 128; the learning rate of the policy network and the evaluation network is 0.001.

[0023] Optionally, the activation function of the policy network and the evaluation network is ReLU; and the loss function of the policy network and the evaluation network is AdamOptimizer.

[0024] Optionally, the CMA-ES algorithm is used to iteratively update the population and state information, specifically comprising:

[0025] initializing population information; the population information at least comprises population size, variable boundary and initial population covariance matrix;

[0026] performing eigenvalue decomposition on the population covariance matrix to obtain eigenvalues and eigenvectors;

[0027] obtaining population individual distribution based on the eigenvalues, the eigenvectors and standard normal distribution, in combination with population mean and population standard deviation;

[0028] randomly sampling in the population individual distribution to obtain a current population;

[0029] based on the Gaussian process surrogate model, evaluating the truss body mass corresponding to each individual in the current population, and selecting the top λ individuals in the current population as population elite individuals; λ is an integer greater than 0;

[0030] based on the current population elite individuals, evolution path, state information and update formula, calculating the population covariance matrix, population mean, population standard deviation and evolution path of the next generation; the update formula at least comprises an evolution path update formula;

[0031] Based on the next generation of population covariance matrix, population mean, population standard deviation and evolutionary path, return and re-execute the step of performing eigenvalue decomposition on the population covariance matrix to obtain eigenvalues and eigenvectors.

[0032] Optionally, the policy network and the evaluation network are trained by reinforcement learning, and specifically comprising:

[0033] According to the state value function and the immediate reward, the advantage function and the action value function are calculated.

[0034] The root mean square error of the action value function and the state value function is taken as the evaluation network loss function, and the evaluation network is trained by back propagation.

[0035] The average value of the advantage function and the importance ratio product is taken as the policy network loss function, and the policy network is trained by back propagation.

[0036] In a second aspect, the present application provides a computer system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the design variable optimization method of fusing reinforcement learning and CMA-ES.

[0037] According to the specific embodiments provided by the present application, the following technical effects are disclosed:

[0038] The design variable optimization method of fusing reinforcement learning and covariance matrix adaptive adjustment evolution strategy (CMA-ES) provided by the present application combines the high-dimensional search capability of CMA-ES algorithm and the sequential decision-making capability of reinforcement learning. By adding a policy network to the evolutionary path update formula of CMA-ES algorithm and training the policy network based on reinforcement learning, the restrictions of inherent rules in the original evolutionary path update formula are broken, the adaptive capability of the policy network is improved, and the generation of a large amount of invalid data is avoided. Therefore, the optimization efficiency of the design variables of the truss body is improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0040] Figure 1A first flowchart of a design variable optimization method that integrates reinforcement learning and CMA-ES, provided in an embodiment of this application;

[0041] Figure 2 The execution flowchart of the CMA-ES algorithm provided in the embodiments of this application;

[0042] Figure 3 A second flowchart of the design variable optimization method that integrates reinforcement learning and CMA-ES provided in the embodiments of this application;

[0043] Figure 4 This is a schematic diagram of the internal structure of a computer system provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] The purpose of this application is to provide a design variable optimization method and system that integrates reinforcement learning and CMA-ES, thereby improving the optimization efficiency of truss body design variables.

[0046] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Example 1

[0048] like Figure 1 As shown, this embodiment provides a design variable optimization method that integrates reinforcement learning and CMA-ES, specifically including:

[0049] Step S1: Determine the truss body design variables, and treat a set of parameters of the truss body design variables as an individual, and treat the set of multiple sets of parameters of the truss body optimization variables as a population; the truss body design variables include at least the side length, thickness and material of the transverse and longitudinal stiffening beams.

[0050] Step S2: Construct a Gaussian process surrogate model, a policy network, and an evaluation network; the Gaussian process surrogate model is used to determine the corresponding truss body mass for each individual; the policy network is used to generate the corresponding evolutionary path change based on the state information of each population; the evaluation network is used to generate the corresponding state value function based on the state information sequence; the state information includes: population mean, population standard deviation, and eigenvalues ​​of the population covariance matrix; the state information sequence includes state information under different iteration numbers.

[0051] As a preferred implementation method, the construction process of the Gaussian process surrogate model specifically includes:

[0052] Step S21: Based on the geometry of the truss body, establish a parametric model according to the design variables of the truss body.

[0053] In this embodiment, the transverse and longitudinal stiffening beams of the load-bearing truss body are selected as the optimization object. The geometry of the load-bearing truss body is simplified, and only the side length, thickness and material selection of the transverse and longitudinal stiffening beams are considered. A parametric model is established based on the simplified geometry.

[0054] Step S22: With the goal of minimizing the mass of the truss body, perform finite element simulation on the parametric model to generate a structural performance dataset; the structural performance dataset includes: multiple sets of parameters of the truss body design variables, and the truss body mass corresponding to each set of parameters.

[0055] In this embodiment, by changing different combinations of side lengths, thicknesses and materials, multiple finite element simulations are performed on the parameter model to obtain multiple sets of parameter data.

[0056] Step S23: Using a set of parameters of the truss body design variables as input and the truss body mass as output, a Gaussian process surrogate model is obtained by fitting the structural performance dataset using a Gaussian process. Details are as follows:

[0057] The first step is to initialize the Gaussian process surrogate model. The input variable X is set as multiple sets of parameters for the truss body design variables, and the output variable y is the truss body mass corresponding to each set of parameters. The radial basis function k(·) is chosen as the kernel function, which determines the smoothness and complexity of the Gaussian process.

[0058] The second step is Gaussian process regression. First, the covariance matrix K of the centralized training samples (i.e., a set of parameters and their corresponding truss body mass) in the structural performance dataset is calculated based on the radial basis function. Then, the model hyperparameter θ is optimized by maximizing the likelihood function logp(y|X,θ). The formula for calculating the likelihood function is:

[0059]

[0060] in, Let I be the noise variance, I be the identity matrix, and n be the number of training samples;

[0061] The third step is to assess and predict the mass of the truss vehicle body. This involves calculating a new set of input parameters x. * The covariance vector k of X * =(x * Using a Gaussian process surrogate model to analyze x), * Prediction is performed to obtain the mean value μ(x) of the output performance index.* ) and variance σ 2 (x * ) :

[0062]

[0063] Fourth step, using the mean μ(x * ) and variance σ 2 (x * ) to construct the new input x * The distribution of the corresponding truss body mass evaluation prediction The predicted value y * is obtained by taking value under the distribution (i.e. the corresponding truss body mass x * ).

[0064] As a preferred embodiment, the constructed hyperparameters in the strategy network and the evaluation network are shown in Table 1:

[0065] Table 1 Hyperparameters of Strategy-Evaluation Network

[0066]

[0067] The role of the strategy network is to generate actions, i.e. to select actions according to the current state. The input of the strategy network is a set of parameters corresponding to the edge length, thickness and material of the beam with horizontal and vertical reinforcement of the current state state; the output is the action action under the current state state, corresponding to the evolution path change.

[0068] The role of the evaluation network is to evaluate the goodness of the action. The evaluation network estimates the value of the combination of the current state and the action. Specifically, the strategy network generates an action through the current state, and then passes the action to the environment (i.e. the process of CMA-ES algorithm), which returns the next state and the immediate reward. The evaluation network receives the next state and the immediate reward, and evaluates the value of the action selection of the strategy network according to them, i.e. the state value function. The difference between the expected value of the immediate reward and the state value evaluation function made by the evaluation network is used to update the parameters of the strategy network and the evaluation network, so that the action selected by the strategy network can obtain higher immediate reward. The input of the evaluation network is the state information sequence under the current optimization trajectory, and the output is the state value function under the current optimization trajectory.

[0069] Step S3: the CMA-ES algorithm is used to iteratively update the population and state information until the set number of iterations is reached, and the state information sequence under the current optimization trajectory is output; the evolution path update formula in the CMA-ES algorithm is added with the strategy network; the process of determining the elite individual in the CMA-ES algorithm applies the Gaussian process proxy model; the state information sequence includes state information under different iteration numbers.

[0070] As shown in Figure 2 , the CMA-ES algorithm is used to iteratively update the population and state information, which specifically includes:

[0071] Step S31: initialize the population information; the population information includes: population size, variable boundary, initial population covariance matrix, and boundary of population standard deviation, etc.

[0072] Step S32: population sampling:

[0073] First, perform eigenvalue decomposition on the population covariance matrix C to obtain eigenvalues D and eigenvectors B; wherein the initial iteration population covariance matrix C is the initial population covariance matrix;

[0074] Second, based on the eigenvalues D, eigenvectors B and standard normal distribution , the population individual distribution x i is obtained in combination with the population mean μ and the population standard deviation σ.

[0075] Third, randomly sample in the population individual distribution , and the sampling number is the population size, so as to obtain the current population.

[0076] Step S33: update each parameter in the CMA-ES algorithm:

[0077] First, based on the Gaussian process surrogate model, evaluate the truss body mass corresponding to each individual in the current population, and select the top λ individuals from the current population as population elite individuals; λ is an integer greater than 0; the population elite individuals are obtained according to the ranking of the truss body mass corresponding to each individual in the current population;

[0078] Second, based on the current population elite individuals, evolution path, state information and update formula, calculate the population covariance matrix, population mean, population standard deviation and evolution path of the next generation; wherein the update formula specifically includes:

[0079] Population mean update formula: In the formula, μ (t+1) is the population mean of the t+1 iteration, μ (t) is the population mean of the t iteration, α μ is the learning rate of the population mean, and α μ ≤1 is used to control the update speed of the population mean, is the i-th population elite individual of the t+1 iteration;

[0080] Evolution path update formula: In the formula, is the evolution path of the t+1th iteration, is the evolution path of the tth iteration, α σ is the weight of the historical evolution path, action is the evolution path change amount corresponding to the state information of the tth iteration of the population output by the policy network, σ (t) is the population standard deviation of the tth iteration;

[0081] Population standard deviation update formula: In the formula, d σ is the standard deviation damping coefficient, which controls the scaling speed of σ, is the expectation of the evolution path of the t+1th iteration, is the mathematical expectation of the L2 Euclidean norm of the random variable subject to the standard normal distribution;

[0082] Population covariance matrix update formula:

[0083]

[0084] In the above formula, part attempts to solve the sign loss problem caused by phenomenon, so another evolution path p c is used to record the sign problem, and the update formula of another evolution path p c is Part of the reference {C λ} all historical data for estimating C, when experienced many generations, using for estimation; α cp is the weight of the historical p c , α cλ , α c1 are the learning rates of the respective parts;

[0085] Step 3: Based on the population covariance matrix, population mean, population standard deviation and evolution path of the next generation, repeat steps S32-S33 until the set number of iterations is reached.

[0086] Step S4: Input the state information sequence under the current optimization trajectory into the evaluation network to obtain the state value function under the current optimization trajectory.

[0087] Step S5: Based on the state information sequence under the current optimization trajectory, the state value function and the immediate reward, the policy network and the evaluation network are trained using reinforcement learning.

[0088] As shown in Figure 3 , the above step S3 will generate a state information sequence S={s1,...,sn}and the reinforcement learning training of the policy network and the evaluation network also needs to reach a set number of optimization trajectory training times to end, so the state information sequence and the state value function under different optimization trajectories are obtained through repeated iteration of the above steps S3-S4, so as to realize the reinforcement learning training of the policy network and the evaluation network.

[0089] In the embodiment, the process of training the policy network and the evaluation network by reinforcement learning includes the following steps:

[0090] Step S51: using the state value function V(s t ) and the immediate reward r t of the current optimization trajectory, calculating the advantage function A(s t ,a t ) and the action value function Q(s t ,a t ).

[0091] The action value function Q(s t ,a t ) represents the expected reward value of performing a specific action a t in a certain state, which is the immediate reward from the current state to the end of the current optimization trajectory and V(s t ), and its calculation formula is Q(s t ,a t ) = r t+1 + γ * V(s t ).

[0092] The advantage function A(s t ,a t ) represents the size difference between the reward value brought by performing a specific action a t in state s t and the expected reward value brought by performing all actions, and its calculation formula is A(s t ,a t ) = Q(s t ,a t )-V(s t ). If the advantage function is positive, it proves that performing a specific action a t will bring positive reward; if the advantage function is negative, it proves that performing a specific action a t will bring negative reward. By using the advantage function as a relative reward function, the model can be better trained.

[0093] Step S52: using the action value function Q(s t ,a tThe root mean square error of the evaluation network is used as the loss function, and the evaluation network is updated through backpropagation. Specifically, the loss function of the evaluation network is...

[0094] Step S53: Sequence the state information S = {s1,...,s} n The evolution path change under the current optimized trajectory is obtained by inputting the policy network and constructing a normal distribution. The log probability density P of the action sequence corresponding to the state information sequence under the normal distribution is then calculated. θ (s t ,a t ); The state sequence S = {s1,...,s} n Input the policy network before the update, obtain the evolution path change under the current optimized trajectory and construct a normal distribution, calculate the log probability density P of the action sequence corresponding to the state information sequence under the normal distribution. θ' (s t ,a t ); with P θ (s t ,a t ) / P θ' (s t ,a t As an importance sample during the training process, it is denoted as the importance ratio. Importance sampling allows the use of samples generated by the old policy to estimate the expected return of the new policy, thus addressing the policy latency problem. To prevent the high variance problem caused by importance sampling, a pruning strategy is used to limit the importance ratio, keeping it within a certain range [1-, 1+], which helps stabilize the learning process and avoids the negative impact of extreme weight values ​​on the learning process.

[0095] Calculate the policy network loss function and backpropagate to update the policy network. Specifically, the policy network loss function is L... actor =mean(min(ratio*advantage,clip(ratio,1-,1+))), where ratio is the importance ratio, advantage is the advantage function, and clip is the clipping strategy.

[0096] Step S6: Determine whether the current optimized trajectory has reached the set number of training iterations. If yes, obtain the reinforcement learning-trained policy network. If not, return to step "Iteratively update the population and state information using the CMA-ES optimization algorithm until the maximum number of iterations is reached, then output the state information sequence under the current optimized trajectory," and re-execute. Figure 3 The process of each step in the process.

[0097] In the embodiment, the evaluation network is mainly used for facilitating the evaluation and training of the policy network. In actual application, the trained policy network is used to assist the population iterative update of the CMA-ES algorithm, and the trained evaluation network is no longer needed.

[0098] Step S7: Based on the trained policy network, the CMA-ES algorithm is used to iteratively update the population and state information until the optimal population is generated after reaching the set number of iterations; the optimal population contains multiple optimal individuals; and the optimal individuals represent a set of optimal parameters of the truss body design variables.

[0099] As a preferred embodiment, the original evolutionary path update formula in the CMA-ES algorithm is wherein, Part of the fixed rules mainly depends on the selection of hyperparameters a σ and λ, which is often low in optimization efficiency when facing complex engineering problems. The main reason is that the original evolutionary path update formula has low adaptability due to the fixed rules.

[0100] To solve this problem, the method of reinforcement learning is used to train the policy network by relying on serialized decision data, which replaces part of the fixed rules in the original evolutionary path update formula, i.e., the evolutionary path update formula is modified to In addition, in the process of training the policy network, a large amount of serialized decision data ensures the applicability of the policy network under different input states. The addition of the policy network breaks through the limitation of the original fixed rules and improves the overall optimization efficiency.

[0101] Step S7 is to determine the action by using the trained policy network of reinforcement learning, and iteratively update the population and state information by using the CMA-ES algorithm again until the optimal population is selected. Because the individuals in the optimal population have been optimized to the maximum, only the optimal individual that meets the requirements needs to be selected from the optimal population. Based on the parameters of the length, thickness and material of the horizontal and vertical beam corresponding to the optimal individual, the truss body structure can be designed, which can ensure that the truss body is the lightest.

[0102] Embodiment 2

[0103] The embodiment provides a computer system which can be a server or a terminal, and an internal structure diagram of the computer system can be as shown in Figure 4As shown in the figure. The computer system includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer system is used to provide computing and control capability. The memory of the computer system includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer system is used to store video tag processing data. The input / output interface of the computer system is used to exchange information between the processor and external devices. The communication interface of the computer system is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to realize a design variable optimization method combining reinforcement learning and CMA-ES.

[0104] Those skilled in the art can understand that, Figure 4 The skilled in the art can understand that,

[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in each embodiment provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

[0106] The database involved in each embodiment provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in each embodiment provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0107] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between each embodiment can be referred to each other.

[0108] The principles and implementation modes of the present application are described by applying specific examples in this paper. The above description of the embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A design variable optimization method that fuses reinforcement learning and CMA-ES, characterized by, The design variable optimization method fusing reinforcement learning and CMA-ES comprises the following steps: determining truss body design variables, taking a set of parameters of the truss body design variables as an individual, and taking a set of multiple sets of parameters of the truss body optimization variables as a population; the truss body design variables at least comprise the side length, thickness and material of the transverse and longitudinal beam; constructing a Gaussian process proxy model, a policy network and an evaluation network; the Gaussian process proxy model is used to determine the corresponding truss body mass according to each individual; the policy network is used to generate the corresponding evolution path change amount according to the state information of each population; the state information comprises the population mean, population standard deviation and eigenvalues of the population covariance matrix; iteratively updating the population and the state information by using the CMA-ES algorithm until a set number of iterations is reached, and then outputting the state information sequence under the current optimization trajectory; the evolution path update formula in the CMA-ES algorithm is added with the policy network; the determination process of the population elite individual in the CMA-ES algorithm applies the Gaussian process proxy model; the state information sequence comprises the state information under different iteration numbers; inputting the state information sequence under the current optimization trajectory into the evaluation network to obtain the state value function under the current optimization trajectory; training the policy network and the evaluation network by using reinforcement learning based on the state information sequence under the current optimization trajectory, the state value function and the immediate reward; determining whether the current optimization trajectory reaches a set number of optimization trajectory training; if yes, a trained policy network is obtained; if no, returning to the step of iteratively updating the population and the state information by using the CMA-ES optimization algorithm until a maximum number of iterations is reached, and then outputting the state information sequence under the current optimization trajectory; iteratively updating the population and the state information by using the CMA-ES algorithm based on the trained policy network until an optimal population is generated after a set number of iterations is reached; the optimal population comprises multiple optimal individuals; the optimal individuals represent a set of optimal parameters of the truss body design variables; wherein iteratively updating the population and the state information by using the CMA-ES algorithm comprises the following steps: initializing population information; the population information at least comprises the population size, variable boundary and initial population covariance matrix; performing eigenvalue decomposition on the population covariance matrix to obtain eigenvalues and eigenvectors; obtaining population individual distribution based on the eigenvalues, the eigenvectors and standard normal distribution, the population mean and the population standard deviation; randomly sampling in the population individual distribution to obtain a current population; evaluating the corresponding truss body mass of each individual in the current population based on the Gaussian process proxy model, and selecting the first λ individuals in the current population as population elite individuals; λ is an integer greater than 0; calculating the population covariance matrix, population mean, population standard deviation and evolution path of the next generation based on the current population elite individuals, evolution path, state information and update formula; the update formula at least comprises an evolution path update formula. Based on the next generation of population covariance matrix, population mean, population standard deviation and evolutionary path, return and re-execute the step "perform eigen decomposition on the population covariance matrix to obtain eigenvalues and eigenvectors". 2.The design variable optimization method of fusing reinforcement learning and CMA-ES according to claim 1, wherein, The evolutionary path update formula is: wherein, is the evolution path of the tth iteration, is the evolution path of the t+1th iteration, a σ is the weight occupied by the historical evolution path, action is the evolution path change amount corresponding to the state information of the tth iteration of the population output by the policy network, μ (t) is the population mean of the tth iteration, μ (t+1) is the population mean of the t+1th iteration, σ (t) is the population standard deviation of the tth iteration. 3.The design variable optimization method of fusing reinforcement learning and CMA-ES according to claim 1, wherein, The construction process of the Gaussian process proxy model specifically includes: Based on the geometric structure of the truss body, a parameter model is established according to the truss body design variables; With the minimum mass of the truss body as the target, finite element simulation is performed on the parameter model to generate a structure performance data set; the structure performance data set includes multiple sets of parameters of the truss body design variables, and the mass of the truss body corresponding to each set of parameters; A Gaussian process is used to fit the structure performance data set to obtain a Gaussian process proxy model, with one set of parameters of the truss body design variables as input and the mass of the truss body as output. 4.The design variable optimization method of fusing reinforcement learning and CMA-ES according to claim 1, wherein, Both the policy network and the evaluation network are provided with two hidden layers, and the number of nodes in each hidden layer is 128; the learning rate of the policy network and the evaluation network is 0.

001. 5.The design variable optimization method of fusing reinforcement learning and CMA-ES according to claim 1, wherein, The activation function of the policy network and the evaluation network is ReLU; the loss function of the policy network and the evaluation network is AdamOptimizer. 6.The design variable optimization method of fusing reinforcement learning and CMA-ES according to claim 1, wherein, The policy network and the evaluation network are trained using reinforcement learning, specifically including: According to the state value function and the immediate reward, the advantage function and the action value function are calculated; The root mean square error of the action value function and the state value function is used as the evaluation network loss function, and the evaluation network is trained by back propagation; The mean value of the minimum value of the product of the advantage function and the importance ratio is used as the policy network loss function, and the policy network is trained by back propagation.

7. A computer system comprising: A memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the design variable optimization method of fusing reinforcement learning and CMA-ES according to any one of claims 1-6.

Images