Bridge cluster multi-agent full-period maintenance decision-making method under deep reinforcement learning

Abstract

The invention discloses a bridge cluster multi-agent full-period maintenance decision-making method under deep reinforcement learning, and the method comprises the steps: firstly dividing a bridge into an upper structure, a lower structure and a bridge deck system, and constructing a simulation environment of bridge cluster maintenance; defining the condition and bridge age of each bridge structure in the bridge cluster; defining three intelligent agents, namely agentsup, agentsub and agentdeck, respectively corresponding to the three part structures, and carrying out training and execution of bridge cluster maintenance; defining a state space and an action space of the deep reinforcement learning method; designing a reward function and determining a state transition matrix; constructing a plurality of agent networks based on a deep Q network algorithm, and training an agent neural network; and finally, testing the capability of the intelligent agent in practical application, and displaying a test result to a user. According to the invention, the structure complexity is greatly simplified, and the method is an important basis for practical application of the maintenance framework.

Description

Technical Field

[0001] The present invention relates to a full-cycle maintenance decision-making method for a bridge cluster multi-agent under deep reinforcement learning, and belongs to the field of structural dynamics and artificial intelligence technology. Background Art

[0002] Bridge cluster maintenance is crucial for ensuring the normal operation of transportation networks, reducing overall maintenance costs, and promoting economic development. Bridge clusters are affected by factors such as traffic loads, environmental erosion, bridge age, and extreme events, which can degrade the overall service level of the transportation network to varying degrees. Bridge management systems can help managers understand bridge operating conditions, predict maintenance needs, and formulate bridge management strategies, thereby maximizing the utilization of bridge assets and extending their service life. Consequently, many countries have developed bridge management systems with their own unique characteristics and functions, such as PONTIS and BRIGIT in the United States, DANBRO in Denmark, SHBMS in South Korea, and BMS in Canada. Degradation models and decision models are key components of these systems. Degradation models predict the performance and lifespan of bridge structures, providing a basis for maintenance decisions throughout their lifecycle. Decision models, based on these predictions, formulate appropriate maintenance plans to optimize resource utilization, extend bridge service life, and improve safety. For bridge clusters, ensuring maintenance strategies are aligned with regional needs and sustainable, they must meet requirements for economic efficiency, safety, reliability, and operational management within varying budgets. By adjusting the maintenance budget, we can balance maintenance costs and maintenance effects within an economically affordable range, ensure timely detection and repair of problems on important bridges, reduce potential risks caused by bridge damage, extend the service life of bridges, and effectively carry out asset management and maintenance planning.

[0003] In traditional decision-making models, dynamic programming and linear programming algorithms are often used to find optimal maintenance strategies. However, due to the large scale of bridge clusters, using these algorithms to solve these problems is often expensive and inefficient. As a machine learning method, reinforcement learning offers numerous advantages, making it highly promising for application in bridge maintenance and providing a novel approach to solving this problem. First, reinforcement learning can solve a variety of maintenance tasks using a simple framework. Unlike traditional methods that require complex models and constraints, reinforcement learning simply defines appropriate states, actions, and reward functions and trains the agent through interaction with the environment to directly learn the optimal maintenance strategy. Furthermore, reinforcement learning can be effectively applied without prior knowledge. By interacting with the environment and learning through trial and error, reinforcement learning can autonomously explore and learn the optimal action strategy. This allows reinforcement learning to learn complex patterns and regularities from data in the maintenance of large bridge clusters, thereby developing maintenance strategies that are adaptable to different scenarios. SUMMARY

[0004] In view of the defects in the prior art, the present application provides a bridge cluster multi-agent full-cycle maintenance decision-making method under deep reinforcement learning. First, the environment of multiple agents and corresponding bridge structures is created, and the agent is constructed for self-learning using a deep Q network (DQN) algorithm. The agent makes decisions by taking the optimal maintenance action according to the current environmental state. After the agent applies the maintenance action to the environment, it obtains environmental feedback and continuously optimizes the maintenance strategy based on the feedback, so that it trains the optimal maintenance strategy suitable for different budgets. During the training process, the feedback of the environment is recorded in numerical form and used as the reward of the agent. By adjusting the proportion of the reward composition elements, the agent trains the maintenance strategy that meets the different budget requirements. For each different bridge structure, the trained agent is stored in the corresponding file. In the testing process, the trained agent is loaded, the framework of multiple agents is constructed, and the interaction environment between multiple agents is constructed. The multi-agent framework makes maintenance decisions based on the state of each bridge structure during the maintenance service life. The state change, maintenance cost and action of each bridge structure are recorded, and the performance of the agent and the effect of the maintenance strategy are analyzed and evaluated through visualization to verify the performance of the agent and the effect of the maintenance strategy.

[0005] The above object is achieved by the following technical solutions:

[0006] The bridge cluster multi-agent full-cycle maintenance decision-making method under deep reinforcement learning of the present application comprises the following steps:

[0007] S1. According to the bridge condition evaluation standard and the bridge operating environment in the bridge cluster, the bridge is divided into three parts: the superstructure, the substructure and the deck system, and a simulation environment for bridge cluster maintenance is constructed. The condition of each bridge structure in the bridge cluster and the bridge age are defined as condition and year. Three agents, agent sup , agent sub and agent deck , are defined, respectively corresponding to the three parts of the structure, for training and executing the bridge cluster maintenance;

[0008] S2. The state space and action space of the deep reinforcement learning method are defined.

[0009] S3. The reward function is designed and the state transition matrix is established.

[0010] S4. A plurality of agent networks based on the deep Q network algorithm is constructed, and the agent neural network is trained.

[0011] S5. The ability of the agent in actual application is tested, and the test results are shown to the user.

[0012] Furthermore, step S1 of defining the condition and age of each bridge structure in the bridge cluster includes the following sub-steps:

[0013] S11: In PyCharm, define the reset function, whose input is the instance variable of the class itself. Its function is to initialize the input of the agent. First, initialize the starting training state of the bridge structure and initialize reward_rating to an empty list. The function output is the initialized states. Then, initialize the component state self.state according to the value of year. When year is zero, the state of all components is zero, indicating the initial state; otherwise, calculate the parameter lam according to the lam baseline value × (serveYears / year), and then use exponential distribution to randomly generate the initial state of each component.

[0014] S12. Define the execute function: its input is the instance variables of the class itself and the maintenance actions taken. According to the state information condition10 in the conditions of the current stage structure and the maintenance actions taken, the corresponding state transition matrix is ​​determined. Then, based on the probability of the state transition matrix, random sampling is performed to determine the state states at the next moment, and the maintenance cost maintenance_cost and the structural risk risk_cost corresponding to the state of the structure in the next stage are determined. The risk_cost is represented by the value of condition10 in the conditions of the bridge structure state information. The corresponding bridge age year is increased by 1 to determine the input nextSta of the next stage of the intelligent agent. te; define the variable Terminal, compare the bridge age year with the maintenance suspension time serveYears. If year>serveYears, then the training ends; otherwise, continue training; finally, add α*maintenance_cost and β*risk_cost to the reward_rating list, where α represents the weight coefficient of maintenance cost, and β represents the weight coefficient of risk cost. α*maintenance_cost represents the weighted value of maintenance cost, which is used to reflect the economic cost of maintenance actions in the reward function; β*risk_cost represents the weighted value of risk cost, which is used to reflect the penalty of risk on the overall strategy in the reward function; output nextState, Terminal: as shown in the formula:

[0015]

[0016] Among them S RLC represents the regional life cycle bridge safety index; s i,trepresents the structural condition level of the i-th bridge in year t, and N represents the number of bridges in the bridge cluster;

[0017] In the training and execution process of the bridge cluster maintenance described in step S1, bridgeMaintenance_singleAgent_training.py is used as the main program, and the main function struAgentTrain implements the iterative training process of the structure corresponding agent. Its input is the structure name stru, the start and end years of training, and the number of training times.

[0018] Furthermore, the steps of defining the state space and action space of the deep reinforcement learning method in step S2 are as follows:

[0019] S21: Define the state input of the agent states = [state sup ,state sub ,state deck ], where state sup 、state sub 、state deck Represent the status of the superstructure, substructure and bridge deck system respectively;

[0020] S22: Define the action output of the agent actions = [action sup ,action sub ,action deck ], where action sup 、action sub 、action deck They represent the maintenance actions of the superstructure, substructure and bridge deck system respectively.

[0021] Furthermore, the reward function design described in step S3 is as follows: the reward r in year i i As shown in formula (1);

[0022] r i =αm i +βf i (1)

[0023] The reward consists of two parts, m i is the maintenance cost of the bridge cluster in year i; f i is the risk cost of the bridge cluster in year i; α represents the weight coefficient of maintenance cost, β represents the weight coefficient of risk cost, and α / β<1, α / β=1, and α / β>1 represent the three types of maintenance strategies: economic priority, balance priority, and safety priority, respectively;

[0024] Maintenance cost m i The calculation process is shown in formula (2):

[0025]

[0026] wherein N is the total number of bridges; is the coefficient corresponding to the bridge structure maintenance action, wherein the coefficients of non-maintenance, small maintenance and reconstruction action are 0, 0.2 and 1.0 respectively; is the coefficient corresponding to the bridge structure condition level, wherein the coefficients corresponding to the structure condition levels from 1 to 5 are 0.4, 0.6, 0.8, 0.9 and 1.0 respectively; l n is the span coefficient of the nth bridge, wherein the bridge span ranges from 10 to 1000 meters, and the corresponding coefficient ranges from 0.8 to 1.0, and the coefficients of different span bridges are obtained by linear interpolation method; m n is the reconstruction cost of the bridge structure; the reward functions of the agents are consistent in span, and only m n are different;

[0027] The calculation process of the risk cost f i is shown in formula (3):

[0028]

[0029] wherein, is the structure condition level of the bridge in the ith year;

[0030] The state transition matrix in step S3 includes a structure degradation state transition probability matrix, a maintenance action state transition probability matrix and a replacement action state transition probability matrix, which are as follows respectively:

[0031] The structure degradation state transition probability matrix is:

[0032] The maintenance action state transition probability matrix is:

[0033] The replacement action state transition probability matrix is:

[0034] Further, the plurality of agent networks based on the deep Q network algorithm in step S4 are constructed, and the agent neural network is trained, and the specific steps are as follows:

[0035] S41: constructing three types of structure agents agent sup , agent sub and agent deckThe network model is used to create an interactive environment and intelligent agents. The DQN single-agent algorithm framework is used to determine the neural network structure for fitting Q values, including a 5-layer convolutional neural network and a 5-layer transposed convolutional neural network. The input shape of each intelligent agent is n×11, the output shape is n×3, the convolution kernel size is 3*3, the span is 1, and the activation function is the ReLU function. The interactive environment is created by calling the class in struDeterioration.py in the environments module of Tensorforce. The intelligent agent is built through the agents module, and the definitions of input states and output actions are consistent with the environment. Initialize the agent sup 、agent sub and agent deck The network parameters are θ sup ,θ sub and θ deck ;

[0036] S42: Construct a target action network model with the same structure as the deep Q network, and let the target action network parameter θ′ sup =θ sup ,θ′ sub =θ sub and θ′ deck =θ deck Use the Environment and Agent classes in the Tensorforce library to create the environment object compEnv and the agent object compAgent. The environment object is created through the Environment.create() function, specifying the struDeterioration environment and passing in the parameters comp and serveYears to simulate the degradation process of the bridge structure. The agent object is created through the Agent.create() function and uses the PPO (Proximal Policy Optimization) algorithm to train the reinforcement learning agent.

[0037] S43: There are two Q networks in the DQN algorithm, namely the current Q network and the target Q network. The update method of the current Q network in DQN is constructed as follows:

[0038] Q(S t ,A t )←Q(S t ,A t )+α[R t+1 +γmaxQ(S t+1 ,a)-Q(S t ,A t )] (4)

[0039] Where, Q(S tA t represents the Q value of the current Q network; a represents the learning rate; S t represents the current state; A t represents the current action; R t+1 represents the immediate reward; g represents the discount factor; maxQ(S t+1 ,a) represents the maximum Q value of the next state S t+1 and action a predicted by the target Q network;

[0040] The training process is visualized using the tqdm module and the tensorboard module in Python, the training progress is displayed, the return value of each training cycle is recorded, and the array episodeReturns is initialized to store the return value; the training is performed by iterating episodeNum times, and each training cycle represents a complete process of the model interacting with the environment;

[0041] S44: The current parameters are updated multiple times at each time step in the current Q network, and are copied into the old parameters of the target Q network after N iterations, and the mean square error of the old parameters DQN is minimized by optimizing the following formula (5) in each update iteration:

[0042]

[0043] In the formula, L i (θ i ) is the loss function; 0 i represents the parameters of the current Q network; E[·] represents the expected value; r represents the immediate reward; g represents the discount factor; represents the parameters of the target Q network predicted by the next state s' and action a' of the maximum Q value; Q(s,a; 0 i ) represents the parameters 0 i predicted by the current Q network of the current state s and action a Q value;

[0044] Specifically, the iterative training is realized through a for loop (the number of loops is defined by the user), the starting year is specified by the user or determined by random sampling from a normal distribution, the reset function in struDeterioration.py is called to initialize the input of the agent, and the Terminal value is reset to False, and a while loop is nested in the for loop to iterate;

[0045] S45: According to the greedy algorithm, the maximum Q value is calculated, as shown in formula (6):

[0046] Q * (s,a)=argmax Q π (s,a) (6)

[0047] Where Q * (s,a) represents the optimal action; argmax Q π (s,a) represents the maximum Q value of the current state s and action a; s represents the state; a represents the action;

[0048] The following modules are executed in each iteration: the decision module calls the agent's act module, outputs actions based on the input states, and initially makes random decisions that gradually become deterministic as training progresses; the execution module calls the execute function in struDeterioration.py to apply the maintenance actions to the corresponding structure, obtaining the states, terminal state, and reward_rating for the next stage; the observation module determines the feedback reward based on the costs calculated by the execute function;

[0049] S46: Sample gradient g of loss by stochastic gradient descent (Sgd) algorithm i Update the Q value in the direction of , as shown in formula (7):

[0050]

[0051] In the formula, θ represents the parameters of the current network; η represents the learning rate; represents the gradient of the loss function;

[0052] Call the observe module of the agent and update the Q value through gradient descent. The update starts after the amount of data in the experience replay pool reaches the starting training value; store the reward in the specified memory unit and repeat the while loop until Terminal = True. When the number of for loop iterations reaches the user-set value, the training ends and the trained agent model data is saved.

[0053] Furthermore, the agent testing process in step S5 has the following aspects:

[0054] S51: The decision module uses a for loop to extract the states of each bridge structure in the bridge cluster according to the structure type, records them in states_stru as the input of the agent, calls the agent of the corresponding structure type in agentList to make a decision and output actions, and adds them to the variable actions_stru that records the overall maintenance actions of the bridge cluster in order, and writes the state of each bridge structure into the variable prevStates_env;

[0055] S52: The execution module calls the execute function in bridgeDeterioration.py to perform the corresponding maintenance action on each bridge structure, obtains nextStates as the input of the next iteration agent; obtains terminal_env and determines whether the structure has reached its service life; obtains reward_rating to calculate the feedback reward of the bridge structure after the maintenance action is taken;

[0056] S53: The recording module calculates the reward_member for each bridge structure after the maintenance action is taken and records the test data to the corresponding location in recordList_episodeTemp, greatly facilitating later data analysis and visualization. The reward_yearSum sum of all bridge structures in a step bridge cluster is then calculated. This sum is then combined and written to episodeReturns for all bridge structures within a year.

[0057] After the while loop is completed, the concat function is called to concatenate the recorded two-dimensional array recordList_episodeTemp, and the time for one iteration of the for loop is output. Finally, the data recorded for every 100 episodes is written to the corresponding file.

[0058] The test results are visualized in bridgeMaintenance_plotting.py by calling the matplotlib module in Python. It imports the relevant data in the file, draws bar charts and line charts, and displays the test results to the user.

[0059] Beneficial effects:

[0060] Compared with the prior art, the present invention has the following advantages:

[0061] First, this invention can simply provide maintenance strategies for bridge clusters under different budgets. The trained strategies will meet the manager's budget requirements, meaning that maintenance strategies can fully utilize budget resources. Second, this method fully utilizes historical inspection data from bridge clusters, effectively utilizing this data to provide valuable information for maintenance decision-making. Furthermore, when using multi-agent approaches to bridge cluster structural maintenance, the neural network structure is relatively simple, requires minimal data volume, and converges quickly.

[0062] 2、The multi-agent framework provided by the application can be used as a general solution for bridge cluster maintenance decision-making, and has superior performance in case migration. The overall structure of the framework remains unchanged, and only a small number of parameters need to be adjusted to apply it to different structure maintenance decision-making. The multi-agent framework can optimize the maintenance strategy based on historical maintenance data after machine simulation training, and can also converge from a random strategy to an optimal maintenance strategy that adapts to the current environment without prior knowledge.

[0063] 3、When dealing with complex bridge clusters, compared with directly increasing network parameters, the application regards structures with similar environmental characteristics and functional characteristics as a whole based on engineering knowledge, and uses an agent to make decisions, greatly simplifying the complexity of the structure, which is an important basis for the practical application of the maintenance framework.

[0064] 4、The application can balance maintenance cost and maintenance effect within an economically affordable range by adjusting the maintenance budget. If the budget increases, more resources can be invested in maintenance work to improve the level of bridge repair and maintenance. If the budget decreases, resource allocation needs to be optimized to ensure that critical repair work is handled first. Reasonable adjustment of the bridge maintenance budget can improve economic efficiency, ensure bridge safety, increase bridge reliability, and achieve effective asset management and maintenance planning.

[0065] 5. To cope with the influence of traffic load, environmental erosion, bridge age and extreme event factors on the performance degradation of bridge clusters, the application develops a bridge cluster life cycle intelligent maintenance management framework by integrating reinforcement learning and deep learning technologies. The framework includes a degradation model framework and an agent maintenance decision-making framework. Based on historical maintenance data and machine learning data, the framework has strong self-learning ability. The agent can develop a maintenance plan that adapts to different budgets according to the degradation of the bridge structure and the current environmental information, to maximize the cost-effectiveness of the bridge cluster maintenance strategy and achieve a balance between risk and economic cost. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 is a whole program flowchart;

[0067] Figure 2 is a single-agent neural network structure;

[0068] Figure 3 is an agent training flowchart;

[0069] Figure 4 is an agent update flowchart;

[0070] Figure 5 is an agent test flowchart;

[0071] Figure 6This is a structural diagram of the multi-agent maintenance framework. DETAILED DESCRIPTION

[0072] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0073] The bridge cluster multi-agent full-cycle maintenance decision-making method based on deep reinforcement learning of the present invention includes the following steps:

[0074] A multi-agent full-cycle maintenance decision-making method for bridge clusters based on deep reinforcement learning, characterized in that the method comprises the following steps:

[0075] S1. According to the bridge condition assessment standards and bridge operation environment in the bridge cluster, the bridge is divided into three parts: superstructure, substructure, and bridge deck system, and a simulation environment for bridge cluster maintenance is constructed; the condition and age of each bridge structure in the bridge cluster are defined; three intelligent agents are defined, namely agent sup 、agent sub and agent deck , respectively corresponding to the three parts of the structure, to carry out the training and execution of bridge cluster maintenance;

[0076] Defining the condition and age of each bridge structure in the bridge cluster includes the following sub-steps:

[0077] S11: In PyCharm, define a reset function whose input is the instance variable of the class itself. This function initializes the agent's input. The purpose of this initialization is to ensure that each component has a reasonable initial state during the simulation and that consistency and repeatability are maintained each time the system is reset. This allows for better control and observation of system behavior during the simulation, facilitating subsequent analysis and optimization. First, initialize the starting training state of the bridge structure (for example, the state of a newly constructed bridge structure is {"bridgen":{"conditions":[0,0,0,0,0,0,0,0,0],"year":[0]}}). Also, initialize reward_rating to an empty list. The function output is the initialized states. Then, initialize the component state self.state based on the value of year. When year is zero, all component states are zero, indicating the initial state. Otherwise, calculate the parameter lam based on the lam baseline value × (serveYears / year). Then, use an exponential distribution to randomly generate the initial state of each component.

[0078] S12. Define the execute function: its input is the instance variables of the class itself and the maintenance actions taken. According to the state information condition10 in the conditions of the current stage structure and the maintenance actions taken, the corresponding state transition matrix is ​​determined. Then, based on the probability of the state transition matrix, random sampling is performed to determine the state states at the next moment, and the maintenance cost maintenance_cost and the structural risk risk_cost corresponding to the state of the structure in the next stage are determined. The risk_cost is represented by the value of condition10 in the conditions of the bridge structure state information. The corresponding bridge age year is increased by 1 to determine the input nextSta of the next stage of the intelligent agent. te; define the variable Terminal, compare the bridge age year with the maintenance suspension time serveYears. If year>serveYears, then the training ends; otherwise, continue training; finally, add α*maintenance_cost and β*risk_cost to the reward_rating list, where α represents the weight coefficient of maintenance cost, and β represents the weight coefficient of risk cost. α*maintenance_cost represents the weighted value of maintenance cost, which is used to reflect the economic cost of maintenance actions in the reward function; β*risk_cost represents the weighted value of risk cost, which is used to reflect the penalty of risk on the overall strategy in the reward function; output nextState, Terminal: as shown in the formula:

[0079]

[0080] Among them S RLC represents the regional life cycle bridge safety index; s i,t represents the structural condition level of the i-th bridge in year t, and N represents the number of bridges in the bridge cluster. The number of years and training times can be adjusted according to the application case.

[0081] After defining the basic conditions of the bridge, we begin training the agent. We use bridgeMaintenance_singleAgent_training.py as the main program. The main function, struAgentTrain, implements the iterative training process for the agent corresponding to the structure. Its input is the structure name, stru, the start and end years of training, and the number of training runs.

[0082] S2. Define the state space and action space of the deep reinforcement learning method; the specific steps are:

[0083] S21: Define the state input of the agent states = [state sup ,state sub,state deck ], where state sup 、state sub 、state deck Representing the states of the superstructure, substructure, and bridge deck, respectively. The states function takes as input the instance variables of the class itself. This function defines the agent's input states, including the structural state information conditions and the bridge age year of the bridge cluster. For example, for the superstructure of a bridge cluster, states = {"bridge0":{conditions,year},…,"bridgen":{conditions,year}}, where conditions = [contition1,…,condition10], where contition1 represents the structural state level in the first year of a decade (i.e., the state level of the structure 10 years ago); contition10 represents the structural state level in the tenth year of a decade (i.e., the structural state level of the current year); and year represents the age of the bridge in the tenth year of a decade. The function outputs the data type, length, and value range.

[0084] S22: Define the action output of the agent actions = [action sup ,action sub ,action deck ], where action sup 、action sub 、action deck They represent the maintenance actions of the superstructure, substructure and bridge deck system respectively.

[0085] S3, reward function design and establishment of state transfer matrix;

[0086] The design of the reward function takes into account the maintenance costs of different bridges under different maintenance categories and the safety of different bridges in different structural states. The individual costs are added together to obtain the total regional cost, and the annual costs are added together to obtain the full life cycle maintenance cost.

[0087] Among them C RLC represents the regional life cycle bridge maintenance cost; N represents the total number of bridges; r s Represents the structural size coefficient; r a Indicates the maintenance action coefficient (the non-action coefficient and maintenance action coefficient are 0 and 0.2 respectively); r c represents the conditional level factor (the coefficients for conditional levels 1 to 5 are set to 0.4, 0.6, 0.8, 0.9, and 1.0); r i,t(s,a) represents the component safety loss coefficient related to the component state s (set to 5).

[0088] The reward function design mentioned above, the reward r in year i i As shown in formula (1);

[0089] r i =αm i +βf i (1)

[0090] The reward consists of two parts, m i is the maintenance cost of the bridge cluster in year i; f i is the risk cost of the bridge cluster in year i; α represents the weight coefficient of maintenance cost, β represents the weight coefficient of risk cost, and α / β<1, α / β=1, and α / β>1 represent the three types of maintenance strategies: economic priority, balance priority, and safety priority, respectively;

[0091] Maintenance cost m i The calculation process is shown in formula (2):

[0092]

[0093] Where N is the total number of bridges; is the coefficient corresponding to the maintenance action of the bridge structure, where the coefficients of no maintenance, minor maintenance, and reconstruction actions are 0, 0.2, and 1.0, respectively; is the coefficient corresponding to the bridge structure condition level. The coefficients corresponding to the structure condition levels from 1 to 5 are 0.4, 0.6, 0.8, 0.9 and 1.0 respectively; l n is the span coefficient of the nth bridge. The span range of bridges is 10 to 1000 meters, and the corresponding coefficient range is 0.8 to 1.0. The coefficients of bridges with different spans are obtained by linear interpolation. n is the reconstruction cost of the bridge structure; the reward function span of each agent is consistent, only m n There is a difference;

[0094] Risk cost f i The calculation process is shown in formula (3):

[0095]

[0096] in, is the structural condition grade of bridge n in year i. In this embodiment, the bridge structure condition grade is divided into 5 grades. A structural condition grade of 1 indicates that the structure is brand new and free of any defects, while a structural condition grade of 5 indicates that the structure has major defects and is impassable.

[0097] The state transition probability in step S3 directly reflects the physical environment in which the decision-making task is to be performed. In the discretized structural state evaluation system, the structural state transition probability exists in the form of a state transition probability matrix. The structural state transition probability matrix is ​​usually obtained from the structural degradation model, experiments and statistical data. In this embodiment, the following three types of maintenance action categories of bridge structure state transition matrices are designed with a time interval of one calendar year. The state transition probability matrix of structural degradation, the state transition probability matrix of maintenance action and the state transition probability matrix of replacement action are as follows:

[0098] State transition probability matrix of structural degradation:

[0099] The state transition probability matrix of the maintenance action:

[0100] The state transition probability matrix of the permutation action:

[0101] S4. Build multiple agent networks based on the deep Q-network algorithm and train the agent neural networks;

[0102] The steps for constructing multiple agent networks based on the deep Q-network algorithm and training the agent neural networks are as follows:

[0103] S41: Constructing three types of structural intelligent agents sup 、agent sub and agent deck The network model is used to create an interactive environment and intelligent agents. The DQN single-agent algorithm framework is used to determine the neural network structure for fitting Q values, including a 5-layer convolutional neural network and a 5-layer transposed convolutional neural network. The input shape of each intelligent agent is n×11, the output shape is n×3, the convolution kernel size is 3*3, the span is 1, and the activation function is the ReLU function. The interactive environment is created by calling the class in struDeterioration.py in the environments module of Tensorforce. The intelligent agent is built through the agents module, and the definitions of input states and output actions are consistent with the environment. Initialize the agent sup 、agent sub and agent deck The network parameters are θ sup ,θ sub and θ deck ;

[0104] S42: Construct a target action network model with the same structure as the deep Q network, and let the target action network parameter θ′ sup =θ sup,θ′ sub =θ sub and θ′ deck =θ deck Use the Environment and Agent classes in the Tensorforce library to create the environment object compEnv and the agent object compAgent. The environment object is created through the Environment.create() function, specifying the struDeterioration environment and passing in the parameters comp and serveYears to simulate the degradation process of the bridge structure. The agent object is created through the Agent.create() function and uses the PPO (Proximal Policy Optimization) algorithm to train the reinforcement learning agent.

[0105] S43: There are two Q networks in the DQN algorithm, namely the current Q network and the target Q network. The update method of the current Q network in DQN is constructed as follows:

[0106] Q(S t ,A t )←Q(S t ,A t )+α[R t+1 +γmaxaQ(S t+1 ,a)-Q(S t ,A t )] (4)

[0107] Where, Q(S t ,A t ) represents the Q value of the current Q network; α represents the learning rate; S t Indicates the current state; A t Indicates the current action; R t+1 represents the immediate reward; γ represents the discount factor; maxQ(S t+1 ,a) represents the next state S predicted by the target Q network t+1 and the maximum Q-value of action a.

[0108] Use the tqdm and tensorboard modules in Python to visualize the training process, display the training progress, record the return value of each training cycle, and initialize the array episodeReturns to store the return value. Execute training by iterating episodeNum times, where each training cycle represents a complete process of interaction between the model and the environment.

[0109] S44: The current parameters are updated multiple times in each time step in the current Q network, and are copied into the old parameters of the target Q network after N iterations. In each update iteration, the mean square error of the old parameters DQN is minimized by optimizing the following formula (5):

[0110]

[0111] In the formula, L i (θ i ) is the loss function; θ i represents the parameters of the current Q network; E[·] represents the expected value; r represents the immediate reward; γ represents the discount factor; represents the parameters of the target Q network the maximum Q value of the predicted next state s' and action a'; Q(s,a; θ i ) represents the parameters θ i of the current Q network

[0112] The iteration training is specifically implemented through a for loop (the number of loops is user-defined training times), the starting year is specified by the user or determined by random sampling through a normal distribution, the reset function in struDeterioration.py is called to initialize the input of the agent, and the Terminal value is reset to False, and a while loop is nested in the for loop for iteration;

[0113] S45: According to the greedy algorithm, the maximum Q value is calculated, as shown in formula (6):

[0114] Q * (s,a)=argmax Q π (s,a) (6)

[0115] In the formula, Q * (s,a) represents the optimal action; argmax Q π (s,a) represents the maximum value of the Q value of the current state s and action a; s represents the state; a represents the action.

[0116] In each iteration, the following modules are executed: the decision module calls the act module of the agent to output actions according to the input states, and the agent makes random decisions in the initial stage and gradually determines as the training progresses; the execution module calls the execute function in struDeterioration.py to apply the maintenance action to the corresponding structure to obtain the states, Terminal state and reward_rating of the next stage; the observation module determines the feedback reward according to the costs calculated by the execute function;

[0117] S46: Sample gradient g of loss by stochastic gradient descent (Sgd) algorithm i Update the Q value in the direction of , as shown in formula (7):

[0118]

[0119] In the formula, θ represents the parameters of the current network; η represents the learning rate; represents the gradient of the loss function.

[0120] Call the observe module of the agent and update the Q value through gradient descent. The update starts after the amount of data in the experience replay pool reaches the starting training value; store the reward in the specified memory unit and repeat the while loop until Terminal = True. When the number of for loop iterations reaches the user-set value, the training ends and the trained agent model data is saved.

[0121] S5. Test the agent's capabilities in actual applications and present the test results to the user. The agent testing process includes the following aspects:

[0122] S51: The decision module uses a for loop to extract the states of each bridge structure in the bridge cluster according to the structure type, records them in states_stru as the input of the agent, calls the agent of the corresponding structure type in agentList to make a decision and output actions, and adds them to the variable actions_stru that records the overall maintenance actions of the bridge cluster in order, and writes the state of each bridge structure into the variable prevStates_env;

[0123] S52: The execution module calls the execute function in bridgeDeterioration.py to perform the corresponding maintenance action on each bridge structure, obtains nextStates as the input of the next iteration agent; obtains terminal_env and determines whether the structure has reached its service life; obtains reward_rating to calculate the feedback reward of the bridge structure after the maintenance action is taken;

[0124] S53: The recording module calculates the reward_member for each bridge structure after the maintenance action is taken and records the test data to the corresponding location in recordList_episodeTemp, greatly facilitating later data analysis and visualization. The reward_yearSum sum of all bridge structures in a step bridge cluster is then calculated. This sum is then combined and written to episodeReturns for all bridge structures within a year.

[0125] After the while loop is completed, the concat function is called to concatenate the recorded two-dimensional array recordList_episodeTemp, and the time for one iteration of the for loop is output. Finally, the data recorded for every 100 episodes is written to the corresponding file.

[0126] The test results are visualized in bridgeMaintenance_plotting.py by calling the matplotlib module in Python. It imports the relevant data in the file, draws bar charts and line charts, and displays the test results to the user.

Claims

1. A multi-agent full-cycle maintenance decision-making method for bridge clusters based on deep reinforcement learning, characterized by: The method comprises the following steps: S1. According to the bridge condition assessment standards and bridge operation environment in the bridge cluster, the bridge is divided into three parts: superstructure, substructure, and bridge deck system, and a simulation environment for bridge cluster maintenance is constructed; the condition and age of each bridge structure in the bridge cluster are defined; three intelligent agents are defined, namely agent sup 、agent sub and agent deck , respectively corresponding to the three parts of the structure, to carry out the training and execution of bridge cluster maintenance; S2. Define the state space and action space of deep reinforcement learning methods; S3, reward function design and establishment of state transfer matrix; S4. Build multiple agent networks based on the deep Q-network algorithm and train the agent neural networks; S5. Test the agent’s capabilities in practical applications and present the test results to the user.

2. The bridge cluster multi-agent full-cycle maintenance decision-making method based on deep reinforcement learning according to claim 1 is characterized in that: Step S1, in which the condition and age of each bridge structure in the bridge cluster are defined, includes the following sub-steps: S11: In PyCharm, define the reset function, whose input is the instance variable of the class itself. Its function is to initialize the input of the agent. First, initialize the starting training state of the bridge structure and initialize reward_rating to an empty list. The function output is the initialized states. Then, initialize the component state self.state according to the value of year. When year is zero, the state of all components is zero, indicating the initial state; otherwise, calculate the parameter lam according to the lam baseline value × (serveYears / year), and then use exponential distribution to randomly generate the initial state of each component. S12: Define the execute function: its input is the instance variables of the class itself and the maintenance actions taken. According to the condition10 in the state information conditions of the current stage structure and the maintenance actions taken, the corresponding state transition matrix is ​​determined. Then, based on the probability of the state transition matrix, random sampling is performed to determine the state states at the next moment, and the maintenance cost maintenance_cost and the structural risk risk_cost corresponding to the state of the structure in the next stage are determined. The risk_cost is represented by the value of condition10 in the bridge structure state information conditions. The corresponding bridge age year is increased by 1 to determine the input nextState of the agent in the next stage. Define the variable Terminal and compare the bridge age year with the maintenance suspension time serveYears. If year > serveYears, the training ends; otherwise, continue training. Finally, add α*maintenance_cost and β*risk_cost to the reward_rating list. α represents the weight coefficient of maintenance cost, and β represents the weight coefficient of risk cost. α*maintenance_cost represents the weighted value of maintenance cost, which is used to reflect the economic cost of maintenance actions in the reward function. β*risk_cost represents the weighted value of risk cost, which is used to reflect the penalty of risk on the overall strategy in the reward function. Output nextState, Terminal: as follows: Among them S RLC represents the regional life cycle bridge safety index; s i,t represents the structural condition level of the i-th bridge in year t, and N represents the number of bridges in the bridge cluster; In the training and execution process of the bridge cluster maintenance described in step S1, bridgeMaintenance_singleAgent_training.py is used as the main program, and the main function struAgentTrain implements the iterative training process of the structure corresponding agent. Its input is the structure name stru, the start and end years of training, and the number of training times.

3. The bridge cluster multi-agent full-cycle maintenance decision-making method based on deep reinforcement learning according to claim 1 is characterized in that: The specific steps of defining the state space and action space of the deep reinforcement learning method in step S2 are as follows: S21: Define the state input of the agent states = [state sup ,state sub ,state deck ], where state sup 、state sub 、state deck Represent the status of the superstructure, substructure and bridge deck system respectively; S22: Define the action output of the agent actions = [action sup ,action sub ,action deck ], where action sup 、action sub 、action deck They represent the maintenance actions of the superstructure, substructure and bridge deck system respectively.

4. The bridge cluster multi-agent full-cycle maintenance decision-making method based on deep reinforcement learning according to claim 1 is characterized in that: The reward function design described in step S3, the reward r in year i i As shown in formula (1); r i =αm i +βf i (1) The reward consists of two parts, m i is the maintenance cost of the bridge cluster in year i; f i is the risk cost of the bridge cluster in year i; α represents the weight coefficient of maintenance cost, β represents the weight coefficient of risk cost, and α / β<1, α / β=1, and α / β>1 represent the three types of maintenance strategies: economic priority, balance priority, and safety priority, respectively; Maintenance cost m i The calculation process is shown in formula (2): In the above formula, N is the total number of bridges; is the coefficient corresponding to the maintenance action of the bridge structure, where the coefficients of no maintenance, minor maintenance, and reconstruction actions are 0, 0.2, and 1.0, respectively; is the coefficient corresponding to the bridge structure condition level. The coefficients corresponding to the structure condition levels from 1 to 5 are 0.4, 0.6, 0.8, 0.9 and 1.0 respectively; l n is the span coefficient of the nth bridge. The span range of bridges is 10 to 1000 meters, and the corresponding coefficient range is 0.8 to 1.

0. The coefficients of bridges with different spans are obtained by linear interpolation. n is the reconstruction cost of the bridge structure; the reward function span of each agent is consistent, only m n There is a difference; Risk cost f i The calculation process is shown in formula (3): in, is the structural condition grade of bridge n in year i; The state transition matrix in step S3 includes the state transition probability matrix of structural degradation, the state transition probability matrix of maintenance action, and the state transition probability matrix of replacement action, which are as follows: State transition probability matrix of structural degradation: The state transition probability matrix of the maintenance action: The state transition probability matrix of the permutation action:

5. The bridge cluster multi-agent full-cycle maintenance decision-making method based on deep reinforcement learning according to claim 1 is characterized in that: Step S4 is to construct multiple agent networks based on the deep Q-network algorithm and train the agent neural networks. The specific steps are as follows: S41: Constructing three types of structural intelligent agents sup 、agent sub and agent deck The network model is used to create an interactive environment and intelligent agents. The DQN single-agent algorithm framework is used to determine the neural network structure for fitting Q values, including a 5-layer convolutional neural network and a 5-layer transposed convolutional neural network. The input shape of each intelligent agent is n×11, the output shape is n×3, the convolution kernel size is 3*3, the span is 1, and the activation function is the ReLU function. The interactive environment is created by calling the class in struDeterioration.py in the environments module of Tensorforce. The intelligent agent is built through the agents module, and the definitions of input states and output actions are consistent with the environment. Initialize the agent sup 、agent sub and agent deck The network parameters are θ sup ,θ sub and θ deck ; S42: Construct a target action network model with the same structure as the deep Q network, and let the target action network parameter θ′ sup =θ sup ,θ′ sub =θ sub and θ′ deck =θ deck ; Use the Environment and Agent classes in the Tensorforce library to create an environment object compEnv and an agent object compAgent. The environment object is created using the Environment.create() function, specifying the struDeterioration environment and passing in the parameters comp and serveYears to simulate the degradation process of the bridge structure. The agent object is created using the Agent.create() function and uses the Proximal Policy Optimization (PPO) algorithm to train the reinforcement learning agent. S43: There are two Q networks in the DQN algorithm, namely the current Q network and the target Q network. The update method of the current Q network in DQN is constructed as follows: Q(S t ,A t )←Q(S t ,A t )+α[R t+1 +γmaxQ(S t+1 ,a)-Q(S t ,A t )] (4) Where, Q(S t ,A t ) represents the Q value of the current Q network; α represents the learning rate; S t Indicates the current state; A t Indicates the current action; R t+1 represents the immediate reward; γ represents the discount factor; maxQ(S t+1 ,a) represents the next state S predicted by the target Q network t+1 and the maximum Q value of action a; Use the tqdm and tensorboard modules in Python to visualize the training process, display the training progress, record the return value of each training cycle, and initialize the array episodeReturns to store the return value. Execute training by iterating episodeNum times, where each training cycle represents a complete process of interaction between the model and the environment. S44: The current parameters are updated multiple times per time step in the current Q network and copied to the old parameters of the target Q network after N iterations. Each update iteration minimizes the mean square error of the old parameters DQN by optimizing the following formula (5): Where, L i (θ i ) is the loss function; θ i represents the parameters of the current Q network; E[·] represents the expected value; r represents the immediate reward; γ represents the discount factor; Represents the parameters of the target Q network The maximum Q value of the predicted next state s' and action a'; Q(s,a;θ i ) represents the parameter θ of the current Q network i The predicted Q value of the current state s and action a; Specifically, iterative training is implemented through a for loop (the number of loops is the number of training times defined by the user). The starting year is specified by the user or determined by random sampling from a normal distribution. The reset function in struDeterioration.py is called to initialize the input of the agent, reset the Terminal value to False, and nest a while loop within the for loop for iteration; S45: Calculate the maximum Q value according to the greedy algorithm, as shown in formula (6): Q * (s,a)=argmax Q π (s,a) (6) Where Q * (s,a) represents the optimal action; argmax Q π (s,a) represents the maximum Q value of the current state s and action a; s represents the state; a represents the action; The following modules are executed in each iteration: the decision module calls the agent's act module, outputs actions based on the input states, and the agent makes random decisions in the initial stage, which gradually become deterministic as training progresses; the execution module calls the execute function in struDeterioration.py, applies the maintenance action to the corresponding structure, and obtains the states, terminal state, and reward_rating of the next stage; The observation module determines the feedback reward based on the costs calculated by the execute function; S46: Sample gradient g of loss by stochastic gradient descent (Sgd) algorithm i Update the Q value in the direction of , as shown in formula (7): In the formula, θ represents the parameters of the current network; η represents the learning rate; represents the gradient of the loss function; Call the observe module of the agent and update the Q value through gradient descent. The update starts after the amount of data in the experience replay pool reaches the starting training value; store the reward in the specified memory unit and repeat the while loop until Terminal = True. When the number of for loop iterations reaches the user-set value, the training ends and the trained agent model data is saved.

6. The bridge cluster multi-agent full-cycle maintenance decision-making method based on deep reinforcement learning according to claim 1 is characterized in that: The agent testing process in step S5 has the following aspects: S51: The decision module uses a for loop to extract the states of each bridge structure in the bridge cluster according to the structure type, records them in states_stru as the input of the agent, calls the agent of the corresponding structure type in agentList to make a decision and output actions, and adds them to the variable actions_stru that records the overall maintenance actions of the bridge cluster in order, and writes the state of each bridge structure into the variable prevStates_env; S52: The execution module calls the execute function in bridgeDeterioration.py to perform the corresponding maintenance action on each bridge structure, obtains nextStates as the input of the next iteration agent; obtains terminal_env and determines whether the structure has reached its service life; obtains reward_rating to calculate the feedback reward of the bridge structure after the maintenance action is taken; S53: The recording module calculates the reward_member for each bridge structure after the maintenance action is taken and records the test data to the corresponding location in recordList_episodeTemp, greatly facilitating later data analysis and visualization. The reward_yearSum sum of all bridge structures in a step bridge cluster is then calculated. This sum is then combined and written to episodeReturns for all bridge structures within a year. After the while loop is completed, the concat function is called to concatenate the recorded two-dimensional array recordList_episodeTemp, and the time for one iteration of the for loop is output. Finally, the data recorded for every 100 episodes is written to the corresponding file. The test results are visualized in bridgeMaintenance_plotting.py by calling the matplotlib module in Python. It imports the relevant data in the file, draws bar charts and line charts, and displays the test results to the user.

Images