Truss optimization method and device based on reinforcement learning, electronic equipment and medium

Abstract

The invention relates to the technical field of buildings, in particular to a reinforcement learning-based truss optimization method and device, electronic equipment and a medium, and the method comprises the steps: obtaining design requirements and historical data of a truss; iteratively searching historical data to obtain a plurality of truss layouts meeting mechanical conditions; and screening the truss layouts of different topologies meeting the design requirements in the plurality of truss layouts, and optimizing the truss layouts of different topologies through a reinforcement learning strategy to obtain an optimization result of the truss. Therefore, the problems that the global optimal result cannot be accurately positioned, the design efficiency is low, the quality of the generated result is poor, the search cost is high and the like due to the adoption of a heuristic search method in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the field of building technology, and in particular to a truss optimization method, device, electronic equipment and medium based on reinforcement learning. Background Art

[0002] Truss structure is a common form of engineering structure, which is composed of rods connected in a certain geometric form. Truss optimization mainly involves the optimization of node positions, topology between nodes and cross-sectional area of ​​connecting rods. Traditionally, structural engineers use sketches to analyze and iterate, which is a cumbersome process and has been gradually replaced by various computer optimization algorithms. However, truss design is a complex combinatorial optimization problem with a huge solution space. It is not feasible to simply use computers for exhaustive search, which requires a lot of time and computing resources.

[0003] In the related technologies, heuristic search methods are basically adopted, which guide the search process by applying some heuristic information or strategies to avoid traversing all possible solutions and find a better solution faster. These methods include genetic algorithms, particle swarm optimization, simulated annealing and differential evolution, as well as improved versions based on them, but the following problems still exist:

[0004] (1) Since the entire search space including node positions and member cross-sectional areas is continuous, slight shape changes may greatly affect the mechanical properties of the entire truss layout. Therefore, if the discretization is not fine enough, it may easily miss the optimal position, resulting in poor quality of the generated results.

[0005] (2) Heuristic search usually only focuses on the objective function and the current decision, while ignoring the global nature of the decision sequence. Since the optimization of the truss structure involves a series of consecutive steps, the decision of the current step can only select the next state, and there is no room for evaluating the new state for further optimization. In the long run, this type of method often cannot find the global optimal result and has low accuracy.

[0006] (3) Using a hierarchical optimization method, the node positions and cross-sectional dimensions are optimized step by step or alternately. This method can increase the search speed, but will gradually reduce the solution space and thus miss the optimal solution. Summary of the invention

[0007] The present application provides a truss optimization method, device, electronic device and storage medium based on reinforcement learning to solve the problems of the inability to accurately locate the global optimal result, low design efficiency, poor quality of generated results, high search cost, etc. caused by the heuristic search method used in the related technology.

[0008] The first aspect of the present application provides a truss optimization method based on reinforcement learning, comprising the following steps: obtaining design requirements and historical data of the truss; iteratively searching the historical data to obtain multiple truss layouts that meet mechanical conditions; screening truss layouts with different topologies that meet the design requirements from the multiple truss layouts, and optimizing the truss layouts with different topologies through a reinforcement learning strategy to obtain an optimization result of the truss.

[0009] Optionally, the optimizing the truss layouts of different topologies by using a reinforcement learning strategy comprises: utilizing a Markov decision model to decide an optimization adjustment action of the truss layout; and utilizing a deep reinforcement learning SAC algorithm to execute the optimization adjustment action to optimize the truss layout.

[0010] Optionally, the Markov decision model includes: a state space, an action space, a reward function, a state transition probability and a discount factor, wherein the state space includes all possible truss layouts, the action space includes all possible adjustment actions, the reward function is used to reflect the quality, stability and degree of constraint satisfaction of the truss layout, the state transition probability reflects the direction of change of the truss layout after performing the adjustment action, and the discount factor is used to balance immediate rewards and future rewards.

[0011] Optionally, the using of a deep reinforcement learning SAC algorithm to perform the optimization adjustment action includes: using a target strategy to map the truss layout to the optimization adjustment action; using an objective function to estimate the value of the truss layout after performing the optimization adjustment action, adding an entropy term to the objective function according to the value, and using the entropy term to balance the target strategy.

[0012] Optionally, the objective function is:

[0013]

[0014] Among them, θ represents the parameters of the target policy π, s t represents the state at time t, a t represents the action at time t, π θ represents the target strategy, which outputs an action a for each state s to maximize the expected cumulative reward, π θ (s t ) indicates state s t The action output by this strategy is π θ (a t |s t ) means in state s t Take action when t The probability of

[0015] The SAC algorithm uses the data buffer D to store all the conversion samples and learns a soft Q function Q with parameter ψ ψ (s t ,a t ), which contains a temperature parameter α to control the degree of exploration and exploitation during the learning process. Indicates state s t The average expectation of the formula when taken from data buffer area D.

[0016] Optionally, the iterative search of the historical data to obtain multiple truss layouts that meet mechanical conditions includes: constructing a search tree for the truss; initializing all adjustment actions on the truss at the beginning of the search, wherein the adjustment action includes one or more of a node adding action, a rod adding action, and a cross-sectional area changing action; iteratively searching the historical design data of the truss based on an upper confidence interval of the tree, wherein in each iterative search process, the search tree is expanded starting from a preset root state of the search tree, and adjustment actions are selected from the preset root state for simulation, and the adjustment actions are updated during the simulation process until the optimal adjustment action is obtained, and multiple truss layouts that meet the mechanical conditions are generated according to the optimal adjustment action of each iteration.

[0017] Optionally, the update formula of the optimal adjustment action is:

[0018] Q(s,a)=βW mean (s,a)+(1-β)W best (s,a)

[0019] Among them, s represents the state of the truss structure, a represents the optimization adjustment action, and W mean (s,a) represents the average reward of all terminal states in the subtree with state s as the root, W best (s, a) represents the highest reward in the subtree, and β is a hyperparameter used to control the exploration tendency between the average reward and the highest reward;

[0020] For each newly generated truss structure G m , define the reward function as:

[0021]

[0022] Among them, K is a scale parameter, Mass( m ) is the mass of the new truss structure, Represents the adjusted reward value. The smaller the mass of the new truss, the higher the reward value.

[0023] The second aspect of the present application provides a truss optimization device based on reinforcement learning, including: an acquisition module, used to acquire the design requirements and historical data of the truss; a search module, used to iteratively search the historical data to obtain multiple truss layouts that meet the mechanical conditions; a screening module, used to screen the truss layouts with different topologies that meet the design requirements in the multiple truss layouts, and optimize the truss layouts with different topologies through a reinforcement learning strategy to obtain the optimization result of the truss.

[0024] Optionally, the screening module is further used to: use a Markov decision model to decide on an optimization adjustment action for the truss layout; and use a deep reinforcement learning SAC algorithm to perform the optimization adjustment action to optimize the truss layout.

[0025] Optionally, the Markov decision model includes: a state space, an action space, a reward function, a state transition probability and a discount factor, wherein the state space includes all possible truss layouts, the action space includes all possible adjustment actions, the reward function is used to reflect the quality, stability and degree of constraint satisfaction of the truss layout, the state transition probability reflects the direction of change of the truss layout after the adjustment action is performed, and the discount factor is used to balance the immediate reward and the future reward.

[0026] Optionally, the screening module is further used to: map the truss layout to the optimization adjustment action using the target strategy; estimate the value of the truss layout after performing the optimization adjustment action using the objective function, add an entropy term to the objective function according to the value, and balance the target strategy using the entropy term.

[0027] Optionally, the objective function is:

[0028]

[0029] Among them, θ represents the parameters of the target policy π, s t represents the state at time t, a t represents the action at time t, π θ represents the target strategy, which outputs an action a for each state s to maximize the expected cumulative reward, π θ (s t ) indicates state s t The action output by this strategy is π θ (a t |s t ) means in state s t Take action when t The probability of

[0030] The SAC algorithm uses the data buffer D to store all the conversion samples and learns a soft Q function Q with parameter ψ ψ (s t ,a t), which contains a temperature parameter α to control the degree of exploration and exploitation during the learning process. Indicates state s t The average expectation of the formula when taken from data buffer area D.

[0031] Optionally, the search module is further used to: construct a search tree for the truss; initialize all adjustment actions on the truss at the beginning of the search, wherein the adjustment action includes one or more of a node addition action, a rod addition action, and a cross-sectional area change action; iteratively search the historical design data of the truss based on the upper confidence interval of the tree, wherein in each iterative search process, the search tree is expanded starting from a preset root state of the search tree, and adjustment actions are selected from the preset root state for simulation, and the adjustment actions are updated during the simulation process until the optimal adjustment action is obtained, and multiple truss layouts that meet mechanical conditions are generated according to the optimal adjustment action of each iteration.

[0032] Optionally, the update formula for the optimal adjustment action is:

[0033] Q(s,a)=βw mean (s, a)+(1-β)W best (s, a)

[0034] Among them, s represents the state of the truss structure, a represents the optimization adjustment action, and W mean (s, a) represents the average reward of all terminal states in the subtree rooted at state s, W best (s, a) represents the highest reward in the subtree, and β is a hyperparameter used to control the exploration tendency between the average reward and the highest reward;

[0035] For each newly generated truss structure G m , define the reward function as:

[0036]

[0037] Among them, K is a scale parameter, Mass(G m ) is the mass of the new truss structure, Indicates the adjusted reward value. The smaller the mass of the new truss, the higher the reward value.

[0038] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the reinforcement learning-based truss optimization method as described in the above embodiment.

[0039] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the reinforcement learning-based truss optimization method as described in the above embodiment.

[0040] Therefore, this application has at least the following beneficial effects:

[0041] The embodiment of the present application can overcome the problems of large search space and high search cost by dividing the search and optimization into two stages, first searching and then fine-tuning the optimization. It can quickly and effectively generate a lightweight truss structure that complies with physical constraints, greatly reduce the manual intervention required in the design process, improve design efficiency, and thus provide a better design solution.

[0042] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0044] Figure 1 A flowchart of a truss optimization method based on reinforcement learning provided according to an embodiment of the present application;

[0045] Figure 2 An example flow chart of the search phase and refinement phase provided according to an embodiment of the present application;

[0046] Figure 3 A pseudo code diagram of the search phase and refinement phase provided according to an embodiment of the present application;

[0047] Figure 4 A schematic diagram of a truss optimization device based on reinforcement learning provided according to an embodiment of the present application;

[0048] Figure 5 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0050] Truss structure is a common form of engineering structure, which is composed of rods connected in a certain geometric form. Truss optimization involves the optimization of node positions, topology between nodes and cross-sectional area of ​​connecting rods. Traditionally, the method of structural engineers to analyze and iterate through sketches is cumbersome and has gradually been replaced by various computer optimization algorithms.

[0051] The related technologies basically adopt the heuristic search method, which guides the search process by applying some heuristic information or strategies to avoid traversing all possible solutions and find a better solution more quickly. However, in the end, it is often easy to miss the optimal position due to the lack of precision in discretization, resulting in poor quality of the generated results; the global optimal solution cannot be found.

[0052] This application analyzes the above problems and proposes the following solutions:

[0053] (1) Although the heuristic search method can generate effective truss layouts, the quality needs to be improved. Reinforcement learning can perform fine optimization of the truss layout, but there are problems such as sparse rewards and difficult training process. Therefore, a two-stage optimization method of searching first and then fine-tuning is adopted. First, a basic solution that meets the mechanical conditions is searched, and then optimization is performed through reinforcement learning on this basis.

[0054] (2) Continuity of search space: The nodes of the truss can be selected at any position in the space, so the optimization action on the node position is continuous; based on the classic tree search, the actions are randomly and uniformly sampled to approximately obtain the optimal action.

[0055] (3) Search result diversity: When the topological structures searched in the first stage are similar, the same optimization results will be obtained after fine-tuning in the second stage, so it is necessary to ensure that the results generated in the search stage have topological diversity. To achieve this goal, in the first stage, not only the optimal truss layout is saved, but also some other suboptimal truss layouts that meet the mechanical conditions but have different topological structures.

[0056] The following describes the reinforcement learning-based truss optimization method, device, electronic device, and storage medium of the embodiment of the present application with reference to the accompanying drawings. Specifically, Figure 1 A schematic flow chart of a truss optimization method based on reinforcement learning provided in an embodiment of the present application.

[0057] like Figure 1 As shown, the truss optimization method based on reinforcement learning includes the following steps:

[0058] In step S101 , the design requirements and historical data of the truss are obtained.

[0059] It is understandable that the embodiments of the present application can obtain the design requirements and historical data of the truss so as to facilitate subsequent iterative search of the historical data to obtain multiple truss layouts that meet the mechanical conditions.

[0060] It should be noted that the design requirements for the truss may include strength, rigidity, stability and good dynamic characteristics, etc., without specific limitation.

[0061] In step S102, historical data is iteratively searched to obtain multiple truss layouts that meet mechanical conditions.

[0062] It is understandable that the embodiments of the present application can obtain multiple truss layouts that meet mechanical conditions by iteratively searching historical data, so as to subsequently screen out truss layouts with different topologies that meet design requirements.

[0063] It should be noted that whether the mechanical conditions are met can be determined by judging whether the current truss layout is stable enough, whether it can meet the load requirements and other conditions. If the current truss layout is stable enough, it meets the mechanical conditions; if the current truss layout meets the load requirements, it meets the mechanical conditions. No specific limitation is made.

[0064] In an embodiment of the present application, historical data is iteratively searched to obtain multiple truss layouts that meet mechanical conditions, including: constructing a search tree for trusses; initializing all adjustment actions on the trusses at the beginning of the search, wherein the adjustment actions include one or more of node addition actions, member addition actions, and cross-sectional area change actions; iteratively searching historical design data of the trusses based on an upper confidence interval of the tree, wherein in each iterative search process, the search tree is expanded starting from a preset root state of the search tree, and adjustment actions are selected from the preset root state for simulation, and the adjustment actions are updated during the simulation process until the optimal adjustment action is obtained, and multiple truss layouts that meet the mechanical conditions are generated according to the optimal adjustment actions of each iteration.

[0065] The preset root state may be determined based on the actual root state of the search tree, without any specific limitation.

[0066] Among them, the update formula of the optimal adjustment action is:

[0067] Q(s,a)=βW mean (s,a)+(1-β)W best (s,a)

[0068] Among them, s represents the state of the truss structure, a represents the optimization adjustment action, and W mean (s,a) represents the average reward of all terminal states in the subtree with state s as the root, W best(s, a) represents the highest reward in the subtree, and β is a hyperparameter used to control the exploration tendency between the average reward and the highest reward;

[0069] For each newly generated truss structure G m , define the reward function as:

[0070]

[0071] Among them, K is a scale parameter, Mass( m ) is the mass of the new truss structure, Indicates the adjusted reward value. The smaller the mass of the new truss, the higher the reward value.

[0072] It can be understood that the embodiment of the present application constructs a search tree and initializes all adjustment actions on the truss at the beginning of the search, iteratively searches the historical design data of the truss based on the upper confidence interval of the tree, and in the iterative search process, selects the adjustment action simulation from the root state of the search tree, and updates the adjustment action until the optimal adjustment action is obtained to generate multiple truss layouts that meet the mechanical conditions, and obtains the optimal or suboptimal effective layout through the search stage, so as to serve as the basis for the subsequent refinement stage.

[0073] Specifically, the goal of the search phase is to find a variety of truss layouts that meet mechanical conditions as the basis for the refinement phase. The process is divided into three steps: adding nodes, adding rods, and changing the cross-sectional area. We use UCT (UpperConfidence Bound applied to Trees) search to guide these three steps to find an effective truss layout. Among them, the UCT algorithm selects the action with the highest UCT value (such as which node to add, which two nodes to connect, which rod cross-sectional area to change, etc.) based on historical data; the specific steps are as follows:

[0074] (1) Initialization: At the beginning of the UCT search, all action values ​​are initialized to 0, which means that the value Q(s, a) of each state-action pair (s, a) is set to 0, where s represents the truss structure state, a represents the optimization action, and Q(s, a) represents the Q function value of the state-action pair (s, a).

[0075] (2) Iterative search: In each UCT iteration, the search process starts from the root state s0 and expands the search tree by selecting the state that makes Q UCT (s, a) the largest action,

[0076] Right now

[0077] Among them, the value of UCT is Q UCT(s, a) consists of two parts, one is the historical average reward, and the other is a term that measures the degree of exploration; by adjusting the weights of these two parts, the degree of exploration and utilization of the system can be controlled.

[0078]

[0079] It is calculated as follows: where Q(s, a) is the action value, c is a constant that controls the balance between exploration and exploitation, n(s) is the number of times state s is visited, and n(s, a) is the number of times action a is executed from state s. Each time a state s is visited, the counters n(s) and n(s, a) are incremented by 1.

[0080] (3) Simulation: Starting from the root state, select actions and simulate until a terminal state is reached. During this process, the counter and action value of each visited state-action pair will be updated; specifically, the value of Q(s, a) will be updated as follows:

[0081] Q(s,a)=βW mean (s, a)+(1-β)W best (s, a)

[0082] Where W mean (s, a) represents the average reward of all terminal states in the subtree rooted at state s, W best (s, a) represents the highest reward in this subtree, and β is a hyperparameter used to control the exploration tendency between the average reward and the highest reward.

[0083] For each newly generated truss structure G m , define the reward function as

[0084]

[0085] Among them, K is a scale parameter, Mass( m ) is the mass of the new truss structure, Indicates the adjusted reward value. The smaller the mass of the new truss, the higher the reward value.

[0086] (4) Limitation of search steps: UCT search will repeat the above process within a given search step budget; once the budget limit is reached, the search process will stop and return the optimal action currently found.

[0087] After each iteration, the system will evaluate whether the current truss layout meets the constraints, such as whether it is stable enough, whether it meets the load requirements, etc. Only layouts that meet the constraints will be considered valid. After the entire search phase is over, the system will save the optimal and some suboptimal valid layouts as the basis for the subsequent refinement phase.

[0088] In step S103, truss layouts with different topologies that meet the design requirements are screened from multiple truss layouts, and the truss layouts with different topologies are optimized by a reinforcement learning strategy to obtain an optimization result of the truss.

[0089] It can be understood that the embodiments of the present application can screen truss layouts of different topologies that meet the design requirements from multiple truss layouts, and optimize the truss layouts of different topologies through reinforcement learning strategies to obtain truss optimization results, thereby quickly and effectively generating lightweight truss structures that comply with physical constraints, overcoming the problems of large search space and high search cost, and avoiding the common problem of falling into local optimality in existing optimization methods, thereby providing a better design solution, greatly reducing the manual intervention required in the design process, and improving design efficiency.

[0090] In an embodiment of the present application, a reinforcement learning strategy is used to optimize truss layouts of different topologies, including: using a Markov decision model to decide on an optimization adjustment action for the truss layout; and using a deep reinforcement learning SAC algorithm to perform the optimization adjustment action to optimize the truss layout.

[0091] The Markov decision model includes: state space, action space, reward function, state transition probability and discount factor; the state space includes all possible truss layouts, the action space includes all possible adjustment actions, the reward function is used to reflect the quality, stability and degree of constraint satisfaction of the truss layout, the state transition probability reflects the direction of change of the truss layout after the adjustment action is performed, and the discount factor is used to balance the immediate reward and future reward.

[0092] It can be understood that the embodiments of the present application can utilize the Markov decision model to decide the optimization adjustment action of the truss layout; utilize the deep reinforcement learning SAC algorithm to perform the optimization adjustment action to optimize the effective truss layout generated in the search phase, thereby quickly and effectively generating a lightweight truss structure that complies with physical constraints.

[0093] It should be noted that the action space includes all possible adjustment actions, such as moving nodes and changing the cross-sectional area of ​​the rod; the reward function is used to reflect the quality of the truss layout, such as the weight of the truss, without specific limitation.

[0094] Specifically, the Markov decision model: the state space S is all possible truss layouts, the action space A includes all possible operations, such as moving nodes and changing the cross-sectional area of ​​the rod; the reward function R is designed to reflect the quality of the truss layout, such as the weight, stability and degree of constraint satisfaction of the truss; the state transition probability P reflects how the layout will change after performing a certain action; the discount factor γ is used to balance the immediate reward and future reward; in order to encourage the generation of lightweight layouts, the following reward function is designed:

[0095]

[0096] Among them, K is a scale parameter, Mass(G m ) is the mass of the new truss structure, represents the value before adjustment, It represents the adjusted value, and the difference is the reward value;

[0097] The lighter the generated layout, the larger the reward; if the layout violates the constraints, such as exceeding the boundaries of the design, or does not meet the stability requirements, a negative reward is given.

[0098] In an embodiment of the present application, a deep reinforcement learning SAC algorithm is used to perform an optimization adjustment action, including: using a target strategy to map the truss layout to the optimization adjustment action; using an objective function to estimate the value of the truss layout after performing the optimization adjustment action, adding an entropy term to the objective function according to the value, and using the entropy term to balance the target strategy.

[0099] Among them, the objective function is:

[0100]

[0101] Among them, θ represents the parameters of the target policy π, s t represents the state at time t, a t represents the action at time t, π θ represents the target strategy, which outputs an action a for each state s to maximize the expected cumulative reward, π θ (s t ) indicates state s t The action output by this strategy is π θ (a t |s t ) means in state s t Take action when t The probability of

[0102] The SAC algorithm uses the data buffer D to store all the conversion samples and learns a soft Q function Q with parameter ψ ψ (s t , a t ), which contains a temperature parameter α to control the degree of exploration and exploitation during the learning process. Indicates state s t The average expectation of the formula when taken from data buffer area D.

[0103] It can be understood that the embodiment of the present application utilizes the target strategy to map the truss layout to the optimization adjustment action, and utilizes the objective function to estimate the value of the truss layout after performing the optimization adjustment action, adds an entropy term to the objective function according to the value, and utilizes the entropy term to balance the target strategy, thereby utilizing the SAC algorithm to optimize the effective truss layout generated in the search phase, so as to quickly and efficiently generate a lightweight truss structure that complies with physical constraints.

[0104] Specifically, the policy π is a neural network model that maps a truss layout (state) to an action; the goal is to find a parameterized policy π that maximizes the expected cumulative reward J(θ):

[0105]

[0106] Among them, s t represents the state at time t, a t represents the action at time t, R(s t , a t ) represents the state-action pair (s t , a t ), γ represents the discount factor, γ t represents the coefficient after t steps of discount, π θ (s t ) represents the strategy π with θ as parameter in s t Actions in state. represents the average expectation when actions are taken from this policy.

[0107] The SAC algorithm uses an actor network (strategy) and a critic network (value function). The actor network generates an action based on the current truss layout, and the critic network estimates the value of the new layout after executing this action. The characteristic of SAC is that an entropy term is added to the objective function. This entropy term encourages the strategy to find a balance between exploration and utilization:

[0108]

[0109] Among them, R(s t ,a t ) represents the state-action pair (s t ,a t ) reward value, π(s t ) indicates that the strategy π is in s t The action in the state, H(π) represents its entropy, α is the temperature parameter, which is used to balance exploration and utilization, E π [·] denotes the expectation under policy π.

[0110] According to the reinforcement learning-based truss optimization method proposed in the embodiment of the present application, the design requirements and historical data of the truss are obtained, and based on the iterative search of the historical data in the search phase, a variety of truss layouts that meet the mechanical conditions are obtained, and the truss layouts with different topologies that meet the design requirements are screened out. The truss layouts with different topologies are optimized through the reinforcement learning strategy to obtain the optimization results of the truss, thereby quickly and effectively generating a lightweight truss structure that complies with physical constraints, overcoming the problems of large search space and high search cost, and being able to avoid falling into local optimality that is common in existing optimization methods, thereby providing a better design solution, greatly reducing the manual intervention required in the design process, and improving design efficiency.

[0111] The following will be combined Figure 2 and Figure 3 The truss optimization method based on reinforcement learning is elaborated in detail. The truss design method of this application is mainly divided into two stages: the search stage and the refinement stage. In the search stage, UCT is used to search and find a variety of effective truss structures; in the refinement optimization stage, the SAC algorithm in deep reinforcement learning is used to further optimize these truss structures, as follows:

[0112] 1. Search phase

[0113] The goal of this stage is to find a variety of truss layouts that meet mechanical conditions as the basis for the refinement stage. The process is divided into three steps: adding nodes, adding rods, and changing the cross-sectional area. The UCT search is used to guide these three steps to find an effective truss layout. The UCT algorithm selects the action with the highest UCT value based on historical data (such as which node to add, which two nodes to connect, which rod cross-sectional area to change, etc.). The specific steps are as follows:

[0114] (1) Initialization: At the beginning of the UCT search, all action values ​​are initialized to 0, which means that the value Q(s,a) of each state-action pair (s,a) is set to 0, where s represents the truss structure state, a represents the optimization action, and Q(s,a) represents the Q function value of the state-action pair (s,a).

[0115] (2) Iterative search: In each UCT iteration, the search process starts from the root state s0 and expands the search tree by selecting the state that makes Q UCT (s,a) the largest action,

[0116] Right now

[0117] Among them, the value of UCT is Q UCT(s,a) consists of two parts, one is the historical average reward, and the other is a term that measures the degree of exploration; by adjusting the weights of these two parts, the degree of exploration and utilization of the system can be controlled.

[0118]

[0119] It is calculated as follows: where Q(s,a) is the action value, c is a constant that controls the balance between exploration and exploitation, n(s) is the number of times state s is visited, and n(s,a) is the number of times action a is executed from state s. Each time a state s is visited, the counters n(s) and n(s,a) are incremented by 1.

[0120] (3) Simulation: Starting from the root state, select actions and simulate until a terminal state is reached. During this process, the counter and action value of each visited state-action pair will be updated; specifically, the value of Q(s,a) will be updated as follows:

[0121] Q(s,a)=βW mean (s,a)+(1-β)W best (s,a)

[0122] Where W mean (s,a) represents the average reward of all terminal states in the subtree with state s as the root, W best (s, a) represents the highest reward in this subtree, and β is a hyperparameter used to control the exploration tendency between the average reward and the highest reward.

[0123] For each newly generated truss structure G m , define the reward function as

[0124]

[0125] Among them, K is a scale parameter, Mass( m ) is the mass of the new truss structure, Represents the adjusted reward value. The smaller the mass of the new truss, the higher the reward value.

[0126] (4) Limitation of search steps: UCT search will repeat the above process within a given search step budget; once the budget limit is reached, the search process will stop and return the optimal action currently found.

[0127] After each iteration, the system will evaluate whether the current truss layout meets the constraints, such as whether it is stable enough, whether it meets the load requirements, etc. Only layouts that meet the constraints will be considered valid. After the entire search phase is over, the system will save the optimal and some suboptimal valid layouts as the basis for the subsequent refinement phase.

[0128] 2. Fine Optimization Stage

[0129] In this phase, the SAC algorithm is used to optimize the effective truss layout generated in the search phase. The reinforcement learning strategy needs to perform two types of actions, namely adjusting the node position and adjusting the cross-sectional area of ​​a specific rod, both of which will be generated by the neural network model.

[0130] (1) Markov decision model (MDP): The state space S is all possible truss layouts, and the action space A includes all possible operations, such as moving nodes and changing the cross-sectional area of ​​the rod. The reward function R is designed to reflect the quality of the truss layout, such as the weight, stability, and degree of constraint satisfaction of the truss. The state transition probability P reflects how the layout will change after executing a certain action. The discount factor γ is used to balance the immediate reward and future reward. In order to encourage the generation of lightweight layouts, the following reward function is designed:

[0131]

[0132] Among them, K is a scale parameter, Mass(G m ) is the mass of the new truss structure, represents the value before adjustment, Represents the adjusted value, and the difference is the reward value.

[0133] The lighter the generated layout, the larger the reward; if the layout violates the constraints, such as exceeding the boundaries of the design, or does not meet the stability requirements, a negative reward is given.

[0134] (2) Strategy and objective function: The policy π is a neural network model that maps a truss layout (state) to an action. The goal is to find a parameterized policy π that maximizes the expected cumulative reward J(θ):

[0135]

[0136] Among them, s t represents the state at time t, a t represents the action at time t, R(s t ,a t ) represents the state-action pair (s t ,a t ), γ represents the discount factor, γ t represents the coefficient after t steps of discount, π θ (s t ) represents the strategy π with θ as parameter in s t Actions in the state, represents the average expectation when actions are taken from this policy.

[0137] The SAC algorithm uses an actor network (strategy) and a critic network (value function). The actor network generates an action based on the current truss layout, while the critic network estimates the value of the new layout after executing this action. The characteristic of SAC is that an entropy term is added to the objective function, which encourages the strategy to find a balance between exploration and exploitation:

[0138]

[0139] Among them, R(s t ,a t ) represents the state-action pair (s t ,a t ) reward value, π(s t ) indicates that the strategy π is in s t The action in the state, H(π) represents its entropy. α is the temperature parameter, which is used to balance exploration and utilization, E π [·] denotes the expectation under policy π.

[0140] (3) Optimization: SAC uses experience replay for training: all state transition samples are saved, and a batch of samples are randomly selected for training to optimize the following objective function:

[0141]

[0142] Among them, θ represents the parameters of the target policy π, s t represents the state at time t, a t represents the action at time t, π θ represents the target strategy, which outputs an action a for each state s to maximize the expected cumulative reward, π θ (s t ) indicates state s t The action output by this strategy is π θ (a t |s t ) means in state s t Take action when t probability.

[0143] The SAC algorithm uses the data buffer D to store all the conversion samples and learns a soft Q function Q with parameter ψ ψ (s t ,a t ), which contains a temperature parameter α to control the degree of exploration and exploitation during the learning process. Indicates state s t The average expectation of the formula when taken from data buffer area D.

[0144] Based on the above method, this application conducted experiments in 2D and 3D test cases. The results showed that AutoTruss improved the average performance of 2D test cases by 6.8%, and in more challenging 3D test cases, the performance was improved by up to 25.1%.

[0145] In summary, the present application can quickly and effectively generate lightweight truss structures that comply with physical constraints, bringing benefits to related fields such as architecture, automobiles and even aerospace; it mainly overcomes the problems of large search space and high search cost by dividing the search and optimization into two stages, searching first and then fine-tuning, and can avoid falling into local optimality that is common in existing optimization methods, thereby providing better design solutions; it can greatly reduce the manual intervention required in the design process and improve design efficiency.

[0146] Next, a reinforcement learning-based truss optimization device proposed according to an embodiment of the present application is described with reference to the accompanying drawings.

[0147] Figure 4 It is a block diagram of a truss optimization device based on reinforcement learning according to an embodiment of the present application.

[0148] like Figure 4 As shown, the reinforcement learning-based truss optimization device 10 includes: an acquisition module 100 , a search module 200 and a screening module 300 .

[0149] Among them, the acquisition module 100 is used to obtain the design requirements and historical data of the truss; the search module 200 is used to iteratively search the historical data to obtain multiple truss layouts that meet the mechanical conditions; the screening module 300 is used to screen the truss layouts with different topologies that meet the design requirements from the multiple truss layouts, and optimize the truss layouts with different topologies through the reinforcement learning strategy to obtain the optimization results of the truss.

[0150] In the embodiment of the present application, the screening module 300 is further used to: use the Markov decision model to decide the optimization adjustment action of the truss layout; use the deep reinforcement learning SAC algorithm to perform the optimization adjustment action to optimize the truss layout.

[0151] In an embodiment of the present application, the Markov decision model includes: a state space, an action space, a reward function, a state transition probability and a discount factor, wherein the state space includes all possible truss layouts, the action space includes all possible adjustment actions, the reward function is used to reflect the quality, stability and degree of constraint satisfaction of the truss layout, the state transition probability reflects the direction of change of the truss layout after the adjustment action is performed, and the discount factor is used to balance the immediate reward and the future reward.

[0152] In an embodiment of the present application, the screening module 300 is further used to: map the truss layout to the optimization adjustment action using the target strategy; estimate the value of the truss layout after performing the optimization adjustment action using the objective function, add an entropy term to the objective function according to the value, and use the entropy term to balance the target strategy.

[0153] In the embodiment of the present application, the objective function is:

[0154]

[0155] Among them, θ represents the parameters of the target policy π, s t represents the state at time t, a t represents the action at time t, π θ represents the target strategy, which outputs an action a for each state s to maximize the expected cumulative reward, π θ (s t ) indicates state s t The action output by this strategy is π θ (a t |s t ) means in state s t Take action when t probability.

[0156] The SAC algorithm uses the data buffer D to store all the conversion samples and learns a soft Q function Q with parameter ψ ψ (s t , a t ), which contains a temperature parameter α to control the degree of exploration and exploitation during the learning process. Indicates state s t The average expectation of the formula when taken from data buffer area D.

[0157] In the embodiment of the present application, the search module 200 is further used to: construct a search tree of trusses;

[0158] Initialize all adjustment actions on the truss at the beginning of the search, wherein the adjustment action includes one or more of a node adding action, a rod adding action, and a cross-sectional area changing action; iteratively search the historical design data of the truss based on the upper confidence interval of the tree, wherein in each iterative search process, the search tree is expanded starting from a preset root state of the search tree, and adjustment actions are selected from the preset root state for simulation, and the adjustment actions are updated during the simulation process until the optimal adjustment action is obtained, and multiple truss layouts that meet mechanical conditions are generated according to the optimal adjustment action of each iteration.

[0159] In the embodiment of the present application, the update formula of the optimal adjustment action is:

[0160] Q(s,a)=β·Wmean (s,a)+(1-P)W best (s, a)

[0161] Among them, s represents the state of the truss structure, a represents the optimization adjustment action, and W mean (s, a) represents the average reward of all terminal states in the subtree rooted at state s, W best (s, a) represents the highest reward in the subtree, and β is a hyperparameter used to control the exploration tendency between the average reward and the highest reward;

[0162] For each newly generated truss structure G m , define the reward function as:

[0163]

[0164] Among them, K is a scale parameter, Mass(G m ) is the mass of the new truss structure, Represents the adjusted reward value. The smaller the mass of the new truss, the higher the reward value.

[0165] It should be noted that the aforementioned explanation of the embodiment of the truss optimization method based on reinforcement learning is also applicable to the truss optimization device based on reinforcement learning in this embodiment, and will not be repeated here.

[0166] According to the reinforcement learning-based truss optimization device proposed in the embodiment of the present application, the design requirements and historical data of the truss are obtained, and the historical data is iteratively searched based on the search phase to obtain diversified truss layouts that meet the mechanical conditions, and the truss layouts with different topologies that meet the design requirements are screened out. The truss layouts with different topologies are optimized through the reinforcement learning strategy to obtain the optimization results of the truss, thereby quickly and effectively generating a lightweight truss structure that complies with physical constraints, overcoming the problems of large search space and high search cost, and being able to avoid falling into local optimality that is common in existing optimization methods, thereby providing a better design solution, greatly reducing the manual intervention required in the design process, and improving design efficiency.

[0167] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0168] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0169] When the processor 502 executes the program, the truss optimization method based on reinforcement learning provided in the above embodiment is implemented.

[0170] Furthermore, the electronic device further comprises:

[0171] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0172] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0173] The memory 501 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0174] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0175] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0176] The processor 502 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0177] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned reinforcement learning-based truss optimization method.

[0178] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0179] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0180] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0181] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0182] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

Claims

1. A truss optimization method based on reinforcement learning, characterized in that: The following steps are involved: Obtain design requirements and historical data for trusses; Iteratively searching the historical data to obtain multiple truss layouts that meet mechanical conditions; The truss layouts of different topologies that meet the design requirements are screened from the multiple truss layouts, and the truss layouts of different topologies are optimized by a reinforcement learning strategy to obtain an optimization result of the truss.

2. The truss optimization method based on reinforcement learning according to claim 1, characterized in that: The method of optimizing the truss layouts of different topologies by using a reinforcement learning strategy comprises: Using a Markov decision model to determine the optimal adjustment action of the truss layout; The optimization adjustment action is performed using a deep reinforcement learning SAC algorithm to optimize the truss layout.

3. The truss optimization method based on reinforcement learning according to claim 2, characterized in that: The Markov decision model includes: a state space, an action space, a reward function, a state transition probability and a discount factor, wherein the state space includes all possible truss layouts, the action space includes all possible adjustment actions, the reward function is used to reflect the quality, stability and degree of constraint satisfaction of the truss layout, the state transition probability reflects the direction of change of the truss layout after the adjustment action is performed, and the discount factor is used to balance the immediate reward and the future reward.

4. The truss optimization method based on reinforcement learning according to claim 2, characterized in that: The using of the deep reinforcement learning SAC algorithm to perform the optimization adjustment action includes: Mapping truss layouts to optimal adjustment actions using target strategies; The objective function is used to estimate the value of the truss layout after the optimization adjustment action is performed, an entropy term is added to the objective function according to the value, and the entropy term is used to balance the target strategy.

5. The truss optimization method based on reinforcement learning according to claim 4, characterized in that: The objective function is: Among them, θ represents the parameters of the target policy π, s t represents the state at time t, a t represents the action at time t, π θ represents the target strategy, which outputs an action a for each state s to maximize the expected cumulative reward, π θ (s t ) indicates state s t The action output by this strategy is π θ (a t |s t ) means in state s t Take action when t probability; The SAC algorithm uses the data buffer D to store all the conversion samples and learns a soft Q function Q with parameter ψ ψ (s t ,a t ), which contains a temperature parameter α to control the degree of exploration and exploitation during the learning process. Indicates state s t The average expectation of the formula when taken from data buffer area D.

6. The truss optimization method based on reinforcement learning according to claim 1, characterized in that: The iterative search of the historical data to obtain a plurality of truss layouts that meet mechanical conditions includes: constructing a search tree of the truss; Initializing all adjustment actions on the truss at the beginning of the search, wherein the adjustment actions include one or more of a node adding action, a member adding action, and a cross-sectional area changing action; The historical design data of the truss is iteratively searched based on the upper confidence interval of the tree, wherein in each iterative search process, the search tree is expanded starting from a preset root state of the search tree, and an adjustment action is selected from the preset root state for simulation, and the adjustment action is updated during the simulation process until the optimal adjustment action is obtained, and multiple truss layouts that meet mechanical conditions are generated according to the optimal adjustment action of each iteration.

7. The truss optimization method based on reinforcement learning according to claim 6, characterized in that: The update formula of the optimal adjustment action is: Q(s,a)=βW mean (s,a)+(1-β)W best (s,a) Among them, s represents the state of the truss structure, a represents the optimization adjustment action, and W mean (s,a) represents the average reward of all terminal states in the subtree with state s as the root, W best (s, a) represents the highest reward in the subtree, and β is a hyperparameter used to control the exploration tendency between the average reward and the highest reward; For each newly generated truss structure G m , define the reward function as: Among them, K is a scale parameter, Mass( m ) is the mass of the new truss structure, Represents the adjusted reward value. The smaller the mass of the new truss, the higher the reward value.

8. A truss optimization device based on reinforcement learning, characterized in that: include: An acquisition module is used to obtain the design requirements and historical data of the trusses; A search module, used for iteratively searching the historical data to obtain multiple truss layouts that meet mechanical conditions; A screening module is used to screen truss layouts of different topologies that meet the design requirements from the multiple truss layouts, and optimize the truss layouts of different topologies through a reinforcement learning strategy to obtain an optimization result of the truss.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the truss optimization method based on reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the truss optimization method based on reinforcement learning as described in any one of claims 1 to 7.

Images