Bridge group maintenance priority dynamic decision-making method and device based on reinforcement learning

Abstract

The invention provides a bridge group maintenance priority dynamic decision-making method and device based on reinforcement learning, and relates to the technical field of bridge intelligent maintenance. The method comprises the following steps: constructing a topological structure of a bridge network and a road; defining a state space, a maintenance action space and a state transition matrix of the bridge; defining a reliability index corresponding to the state of the bridge, and designing a comprehensive reward function based on maintenance cost, asset risk and traffic network capacity loss risk based on the topological structure; constructing a bridge maintenance decision problem; the method comprises the following steps of: describing a bridge maintenance decision problem as a Markov decision process, establishing a pointer network strategy model by adopting a pointer network, and training the pointer network strategy model by adopting an Actor-Critic algorithm to obtain a maintenance decision model based on reinforcement learning; and training the maintenance decision model based on reinforcement learning until convergence, and outputting a bridge maintenance action sequence under limited constraints. By adopting the method, the limitation problem of traditional single bridge assessment can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent bridge maintenance, and in particular to a method and device for dynamic decision-making on maintenance priorities of a group of bridges based on reinforcement learning. Background Art

[0002] Bridges degrade over their service life due to environmental and load conditions. As a vital component of the infrastructure network, bridge maintenance is impacted by funding, environmental, and social factors. Existing bridge management systems primarily optimize budget allocation and develop maintenance strategies, but these efforts face efficiency bottlenecks.

[0003] Currently, most domestic bridge management strategies rely on either periodic or condition-based maintenance. Periodic maintenance strategies rely on predetermined, fixed maintenance cycles, making them inflexible and inadequate for adapting to maintenance needs. Condition-based maintenance strategies employ fixed maintenance strategies based on bridge conditions, but budget constraints prevent timely maintenance of all bridges and prevent the proper allocation of maintenance budgets to critical bridges.

[0004] The existing bridge maintenance decision-making method based on dynamic programming and genetic algorithms is a static optimization method with low intelligence. It lacks consideration of the maintenance decision-making problem of regional bridge groups and has difficulty in handling large-scale state space and action space and long-term non-stationary environment decision-making problems.

[0005] Existing bridge maintenance priority rankings are based on maintenance costs or the subjective experience of decision-makers, making it difficult to ensure optimal decision-making. Furthermore, bridge status affects the service performance of the transportation network, and different bridges exhibit different evolution patterns, leading to variations in bridge maintenance priorities. Existing research has not fully considered the impact of the state evolution of bridge groups on transportation network flow, making it difficult to dynamically prioritize bridge maintenance based on changes in network capacity demand within maintenance budget constraints. Summary of the Invention

[0006] To address the existing problems of low intelligence, lack of consideration for regional bridge group maintenance decision-making, difficulty in handling large-scale state space and action space and long-term non-stationary environment decision-making, and difficulty in implementing dynamic maintenance priority sorting for bridge maintenance changes under budget maintenance funding constraints, the present invention provides a method and device for dynamic decision-making on bridge group maintenance priority based on reinforcement learning. The technical solution is as follows:

[0007] In one aspect, a method for dynamic decision-making on maintenance priorities of a group of bridges based on reinforcement learning is provided. The method is implemented by a dynamic decision-making device for maintenance priorities of a group of bridges based on reinforcement learning. The method includes:

[0008] S1. Construct the topological structure of the bridge network and roads; define the bridge state space, maintenance action space, and state transition matrix;

[0009] S2. Define the reliability index corresponding to the bridge status. Based on the topological structure of the bridge network and roads, design a comprehensive reward function based on maintenance cost, asset risk, and transportation network capacity loss risk.

[0010] S3. Constructing a bridge maintenance decision problem based on the bridge state space, maintenance action space, state transition matrix, and the comprehensive reward function;

[0011] S4. Describe the bridge maintenance decision-making problem as a Markov decision process, use a pointer network to establish a pointer network policy model, and use the actor-critic algorithm to train the pointer network policy model to obtain a maintenance decision model based on reinforcement learning. The maintenance decision model based on reinforcement learning is trained until convergence, and the output is the ranking of bridge maintenance actions under limited constraints.

[0012] Optionally, the state space of the bridge is expressed by the following formula (1):

[0013] (1)

[0014] Where S represents the state space of the bridge;

[0015] The maintenance action space is expressed by the following formula (2):

[0016] (2)

[0017] Among them, A represents the maintenance action space;

[0018] The state transfer matrix is ​​expressed by the following formula (3):

[0019] (3)

[0020] Where P represents the state transfer matrix.

[0021] Optionally, the comprehensive reward function based on maintenance cost, asset risk and traffic network capacity loss risk is expressed by the following formula (4):

[0022] (4)

[0023] in, The asset risk corresponding to the bridge status is expressed by the following formula (5):

[0024] (5)

[0025] in, The risk of traffic capacity loss of the bridge network corresponding to the bridge state is expressed by the following formula (6):

[0026] (6)

[0027] in, represents the comprehensive reward function; represents direct maintenance cost; represents the direct maintenance cost and asset risk scaling factor; represents the risk weight coefficient of bridge network traffic capacity loss; represents the direct maintenance cost and asset risk scaling factor; represents the failure probability corresponding to the bridge state; represents the cost of asset failure; Indicates the bridge failure combination corresponding to the bridge status; Represents the traffic capacity loss of the bridge network under the bridge failure combination.

[0028] Optionally, S4 describes the bridge maintenance decision problem as a Markov decision process, uses a pointer network to establish a pointer network policy model, uses an Actor-Critic algorithm to train the pointer network policy model, and establishes a maintenance decision model based on reinforcement learning; the maintenance decision model based on reinforcement learning is trained until convergence, and a bridge maintenance action ranking under limited constraints is output, including:

[0029] S41. A pointer strategy network model is constructed using a pointer network. Bridge attributes, bridge status, and maintenance actions are input into an encoder to generate feature codes. A decoder is used to integrate the feature codes, current bridge information, and the numbers of traversed bridges. An attention mechanism is used to generate the element with the largest weight as the bridge to be repaired and the bridge maintenance action. Based on the bridges and the bridge maintenance actions, a preliminary maintenance cost is calculated.

[0030] S42. Construct an Actor-Critic network based on the pointer policy network model; wherein the Actor-Critic network includes a pointer policy network and a value network; wherein the pointer policy network is used to output a bridge maintenance priority and a maintenance action strategy based on the bridge status; wherein the value network is used to predict the state value based on the bridge status and the information processed by the encoder;

[0031] S43. Construct an advantage function; based on the advantage function, construct a pointer strategy network loss function and a value network loss function;

[0032] S44. Perform repairs according to the bridge repair priority ranking generated by the pointer policy network to obtain the actual reward value; take the mean square error between the actual reward value and the estimated reward value generated by the value network as the optimization target, and use the gradient descent method to train the Actor-Critic network through the pointer policy network loss function and the value network loss function until convergence, and output the bridge repair action ranking under limited constraints.

[0033] Optionally, the advantage function is expressed by the following formula (7):

[0034] (7)

[0035] Among them, r represents the reward obtained at the current time step; represents the discount factor; Represents the value network's valuation of the next state; Represents the value network's valuation of the current state; is the advantage function, which represents the benefit of performing action a in state s; s represents the current state of the bridge; a represents the maintenance action taken.

[0036] Optionally, the loss function of the pointer strategy network is expressed by the following formula (8):

[0037] (8)

[0038] in, represents the loss of the pointer strategy network; E represents the expected operation; Represents the policy function About its implicit parameters The logarithmic gradient of ; where It represents the probability of taking action a in state s.

[0039] Optionally, the loss function of the value network is expressed by the following formula (9):

[0040] (9)

[0041] in, Represents the loss of the value network.

[0042] On the other hand, a device for dynamic decision-making on maintenance priority of a group of bridges based on reinforcement learning is provided. The device is applied to a method for dynamic decision-making on maintenance priority of a group of bridges based on reinforcement learning. The device includes:

[0043] The first construction unit is used to construct the topological structure of the bridge network and the road; define the state space, maintenance action space and state transition matrix of the bridge;

[0044] A design unit is used to define the reliability index corresponding to the bridge status. Based on the topological structure of the bridge network and the road, a comprehensive reward function is designed based on the maintenance cost, asset risk and the risk of loss of transportation network capacity;

[0045] The second construction unit is used to construct a bridge maintenance decision problem based on the bridge state space, the maintenance action space, the state transition matrix and the comprehensive reward function;

[0046] The training unit is used to describe the bridge maintenance decision-making problem as a Markov decision process, establish a pointer network policy model using a pointer network, train the pointer network policy model using an actor-critic algorithm, and establish a maintenance decision model based on reinforcement learning; the maintenance decision model based on reinforcement learning is trained until convergence, and the bridge maintenance action ranking under limited constraints is output.

[0047] Optionally, the state space of the bridge is expressed by the following formula (1):

[0048] (1)

[0049] Where S represents the state space of the bridge;

[0050] The maintenance action space is expressed by the following formula (2):

[0051] (2)

[0052] Among them, A represents the maintenance action space;

[0053] The state transfer matrix is ​​expressed by the following formula (3):

[0054] (3)

[0055] Where P represents the state transfer matrix.

[0056] Optionally, the comprehensive reward function based on maintenance cost, asset risk and traffic network capacity loss risk is expressed by the following formula (4):

[0057] (4)

[0058] in, The asset risk corresponding to the bridge status is expressed by the following formula (5):

[0059] (5)

[0060] in, The risk of traffic capacity loss of the bridge network corresponding to the bridge state is expressed by the following formula (6):

[0061] (6)

[0062] in, represents the comprehensive reward function; represents direct maintenance cost; represents the direct maintenance cost and asset risk scaling factor; represents the risk weight coefficient of bridge network traffic capacity loss; represents the direct maintenance cost and asset risk scaling factor; represents the failure probability corresponding to the bridge state; represents the cost of asset failure; Indicates the bridge failure combination corresponding to the bridge status; Represents the traffic capacity loss of the bridge network under the bridge failure combination.

[0063] Optionally, the training unit is used to:

[0064] A pointer network is used to construct a pointer policy network model. Bridge attributes, bridge status, and maintenance actions are input into an encoder to generate feature codes. A decoder integrates the feature codes, current bridge information, and the numbers of traversed bridges. An attention mechanism is used to generate the element with the largest weight as the next bridge to be repaired and the bridge maintenance action. Based on the bridges and bridge maintenance actions, a preliminary maintenance cost is calculated.

[0065] Based on the pointer policy network model, an actor-critic network is constructed. The actor-critic network includes a pointer policy network and a value network. The pointer policy network is used to output the bridge maintenance priority and maintenance action strategy based on the bridge status. The value network is used to predict the state value based on the bridge status and the information processed by the encoder.

[0066] Construct an advantage function; based on the advantage function, construct the pointer strategy network loss function and the value network loss function;

[0067] Repairs are performed according to the bridge maintenance priority ranking generated by the pointer policy network to obtain the actual reward value; the mean square error between the actual reward value and the estimated reward value generated by the value network is used as the optimization target, and the actor-critic network is trained using the gradient descent method until convergence through the pointer policy network loss function and the value network loss function, outputting the ranking of bridge maintenance actions under limited constraints.

[0068] Optionally, the advantage function is expressed by the following formula (7):

[0069] (7)

[0070] Among them, r represents the reward obtained at the current time step; represents the discount factor; Represents the value network's valuation of the next state; Represents the value network's valuation of the current state; is the advantage function, which represents the benefit of performing action a in state s; s represents the current state of the bridge; a represents the maintenance action taken.

[0071] Optionally, the loss function of the pointer strategy network is expressed by the following formula (8):

[0072] (8)

[0073] in, represents the loss of the pointer strategy network; E represents the expected operation; Represents the policy function About its implicit parameters The logarithmic gradient of ; where It represents the probability of taking action a in state s.

[0074] Optionally, the loss function of the value network is expressed by the following formula (9):

[0075] (9)

[0076] in, Represents the loss of the value network.

[0077] On the other hand, a bridge group maintenance priority dynamic decision-making device based on reinforcement learning is provided, and the bridge group maintenance priority dynamic decision-making device based on reinforcement learning includes: a processor; a memory, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, any one of the above-mentioned bridge group maintenance priority dynamic decision-making methods based on reinforcement learning is implemented.

[0078] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned reinforcement learning-based dynamic decision-making methods for bridge group maintenance priorities.

[0079] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0080] The embodiment of the present invention first constructs the topological structure of the bridge network and the road; defines the state space, maintenance action space and state transition matrix of the bridge; defines the reliability index corresponding to the state of the bridge, and designs a comprehensive reward function based on maintenance cost, asset risk and traffic network capacity loss risk; secondly, constructs a bridge maintenance decision problem based on the state space, maintenance action space, state transition matrix and comprehensive reward function of the bridge; finally, describes the bridge maintenance decision problem as a Markov decision process, uses a pointer network to establish a pointer network policy model, uses the actor-critic algorithm to train the pointer network policy model, and establishes a maintenance decision model based on reinforcement learning; trains the maintenance decision model based on reinforcement learning until convergence, and outputs a ranking of bridge maintenance actions under limited constraints.

[0081] By constructing a maintenance decision model based on reinforcement learning, the embodiments of the present invention can accurately simulate the degradation state of bridges and provide reliable data for subsequent decision-making. By constructing the topological structure of bridge networks and roads, the influence of bridge status on the risk of traffic network flow loss is calculated, and the impact of maintenance plans on maintenance costs, asset risks and traffic network flow loss risks is calculated, further addressing the limitations of traditional single bridge assessments. Through reinforcement learning and pointer networks, the embodiments of the present invention determine the importance of bridges in affecting traffic network flow, and then give a bridge maintenance priority ranking. Under the constraints of limited maintenance budgets, better large-scale bridge group maintenance decisions can be made, and the traffic network performance can be improved by efficiently utilizing maintenance resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0083] Figure 1 This is a flow chart of a method for dynamic decision-making on bridge group maintenance priority based on reinforcement learning provided by an embodiment of the present invention;

[0084] Figure 2 1 is a schematic diagram of a topological structure of a bridge network provided by an embodiment of the present invention;

[0085] Figure 3 This is a schematic diagram of maintenance priority sorting based on reinforcement learning provided by an embodiment of the present invention;

[0086] Figure 4 This is a distribution diagram of the natural degradation state of a bridge provided by an embodiment of the present invention;

[0087] Figure 5This is a block diagram of a dynamic decision-making device for bridge group maintenance priority based on reinforcement learning provided by an embodiment of the present invention;

[0088] Figure 6 This is a structural diagram of a reinforcement learning-based dynamic decision-making device for bridge group maintenance priorities provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0089] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0090] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0091] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0092] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0093] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0094] The embodiment of the present invention provides a method for dynamic decision-making of bridge group maintenance priority based on reinforcement learning. The method can be implemented by a dynamic decision-making device for bridge group maintenance priority based on reinforcement learning. The dynamic decision-making device for bridge group maintenance priority based on reinforcement learning can be a terminal or a server. Figure 1 The flowchart of the dynamic decision-making method for bridge group maintenance priority based on reinforcement learning is shown. The processing flow of this method may include the following steps:

[0095] S1. Construct the topological structure of the bridge network and roads; define the bridge state space, maintenance action space, and state transition matrix.

[0096] In a feasible implementation, Figure 2The figure shows a schematic diagram of the topological structure of the bridge network provided by an embodiment of the present invention; wherein the bridge network is abstracted into a topological result form of a directed graph, which can be expressed as ; Wherein, G represents a directed graph, each node N in the directed graph represents a road intersection; the node information includes the intersection location coordinates; each edge E of the directed graph represents the connecting section between intersections; the edge information includes OD traffic flow and bridge information.

[0097] Optionally, the state space of the bridge is represented by the following formula (1):

[0098] (1)

[0099] Where S represents the state space of the bridge;

[0100] The maintenance action space is expressed by the following formula (2):

[0101] (2)

[0102] Among them, A represents the maintenance action space;

[0103] The state transfer matrix is ​​expressed by the following formula (3):

[0104] (3)

[0105] Where P represents the state transfer matrix.

[0106] Among them, the state transfer matrix includes five state transfer matrices corresponding to different maintenance actions, and the dimension of each action state transfer matrix is , if the state of the bridge at time t is expressed as , take any maintenance action Finally, the state of the bridge is based on the state transition matrix The probability distribution is sampled and becomes .

[0107] S2. Define the reliability index corresponding to the bridge status. Based on the topological structure of the bridge network and roads, design a comprehensive reward function based on maintenance cost, asset risk and traffic network capacity loss risk.

[0108] Alternatively, the comprehensive reward function based on maintenance cost, asset risk and transportation network capacity loss risk is expressed by the following formula (4):

[0109] (4)

[0110] in, The asset risk corresponding to the bridge status is expressed by the following formula (5):

[0111] (5)

[0112] in, The risk of traffic capacity loss of the bridge network corresponding to the bridge state is expressed by the following formula (6):

[0113] (6)

[0114] in, represents the comprehensive reward function; represents direct maintenance cost; represents the direct maintenance cost and asset risk scaling factor; represents the risk weight coefficient of bridge network traffic capacity loss; represents the direct maintenance cost and asset risk scaling factor; represents the failure probability corresponding to the bridge state; represents the cost of asset failure; Indicates the bridge failure combination corresponding to the bridge status; Represents the traffic capacity loss of the bridge network under the bridge failure combination.

[0115] S3. Construct a bridge maintenance decision problem based on the bridge state space, maintenance action space, state transition matrix, and comprehensive reward function.

[0116] Among them, the Markov decision process includes a five-tuple that can be expressed as: (S, A, P, R, γ); among them, S represents; A represents; P represents; R represents the comprehensive reward function; γ represents the discount factor, which takes a value of [0, 1] and is used to balance the relationship between current returns and future returns. The lower the value, the more emphasis will be placed on current rewards, focusing on short-term cost savings, and vice versa, emphasizing long-term benefits.

[0117] S4. Describe the bridge maintenance decision-making problem as a Markov decision process, use a pointer network to establish a pointer network policy model, and use the actor-critic algorithm to train the pointer network policy model to obtain a maintenance decision model based on reinforcement learning. The maintenance decision model based on reinforcement learning is trained until convergence, and the output is the ranking of bridge maintenance actions under limited constraints.

[0118] Among them, the pointer network is a sequence-to-sequence learning model that can solve the problem that the traditional seq2seq model cannot handle through the attention mechanism, that is, the output sequence changes as the length of the input sequence changes.

[0119] Among them, Figure 3The figure shows a schematic diagram of maintenance priority sorting based on reinforcement learning provided by an embodiment of the present invention. In a feasible implementation, the bridge attributes, the current state of the bridge, and the maintenance actions that can be taken are input into an encoder to generate feature codes, and the feature codes are input into a decoder. The element with the largest weight is generated through an attention mechanism as the bridge to be repaired next and the maintenance action to be taken for the bridge. The preliminary maintenance cost is calculated based on the bridge currently being repaired and the maintenance action taken for the bridge. When the maintenance budget is sufficient, the maintenance is agreed and the maintenance action is completed; when the maintenance budget is insufficient, the maintenance is rejected.

[0120] Optionally, the specific implementation process of S4 includes S41-S414:

[0121] S41. A pointer strategy network model is constructed using a pointer network. Bridge attributes, the current state of the bridge, and the available maintenance actions are input into an encoder to generate feature codes. A decoder is used to integrate the feature codes, the current bridge information, and the numbers of the traversed bridges. The attention mechanism is used to generate the bridge with the largest weight and the corresponding selected action as the next bridge to be repaired and the maintenance action to be taken for the bridge. A preliminary maintenance cost is calculated based on the bridges to be repaired and the maintenance actions taken for the bridge.

[0122] Among them, in each step, according to the attention mechanism, the bridge with the largest weight is taken as the final output.

[0123] Among them, when the maintenance budget is sufficient, the maintenance action is completed; when the maintenance budget is insufficient, the maintenance action is rejected and no maintenance is performed.

[0124] S42. Construct an Actor-Critic network based on the pointer policy network model; wherein the Actor-Critic network includes a pointer policy network and a value network; wherein the pointer policy network is used to output a bridge maintenance priority and a maintenance action strategy based on the bridge status; wherein the value network is used to predict the state value based on the bridge status and the information processed by the encoder;

[0125] S43. Construct an advantage function; based on the advantage function, construct a pointer strategy network loss function and a value network loss function;

[0126] Optionally, the advantage function is expressed by the following formula (7):

[0127] (7)

[0128] Among them, r represents the reward obtained at the current time step; represents the discount factor; Represents the value network's valuation of the next state; Represents the value network's valuation of the current state; is the advantage function, which represents the benefit of performing action a in state s; s represents the current state of the bridge; a represents the maintenance action taken.

[0129] Optionally, the loss function of the pointer policy network is expressed by the following formula (8):

[0130] (8)

[0131] in, represents the loss of the pointer strategy network; E represents the expected operation; Represents the policy function About its implicit parameters The logarithmic gradient of ; where It represents the probability of taking action a in state s.

[0132] Optionally, the loss function of the value network is expressed by the following formula (9):

[0133] (9)

[0134] in, Represents the loss of the value network.

[0135] S44. Perform repairs according to the bridge repair priority ranking generated by the pointer policy network to obtain the actual reward value; take the mean square error between the actual reward value and the estimated reward value generated by the value network as the optimization target, and use the gradient descent method to train the Actor-Critic network through the pointer policy network loss function and the value network loss function until convergence, and output the bridge repair action ranking under limited constraints.

[0136] Among them, by adopting the embodiment of the present invention, a maintenance strategy with lower maintenance costs for bridge groups and lower risks of traffic network capacity loss can be implemented.

[0137] Among them, such as Figure 4 The figure shows a distribution diagram of the natural degradation state of a bridge provided by an embodiment of the present invention; wherein the state degrees are divided into state 0, state 1, state 2, state 3 and state 4.

[0138] The embodiment of the present invention first constructs the topological structure of the bridge network and the road; defines the state space, maintenance action space and state transition matrix of the bridge; defines the reliability index corresponding to the state of the bridge, and designs a comprehensive reward function based on maintenance cost, asset risk and traffic network capacity loss risk; secondly, constructs a bridge maintenance decision problem based on the state space, maintenance action space, state transition matrix and comprehensive reward function of the bridge; finally, describes the bridge maintenance decision problem as a Markov decision process, uses a pointer network to establish a pointer network policy model, uses the actor-critic algorithm to train the pointer network policy model, and establishes a maintenance decision model based on reinforcement learning; trains the maintenance decision model based on reinforcement learning until convergence, and outputs a ranking of bridge maintenance actions under limited constraints.

[0139] By constructing a maintenance decision model based on reinforcement learning, the embodiments of the present invention can accurately simulate the degradation of bridge status and provide reliable data for subsequent decision-making. By constructing the impact law of traffic network flow loss, the impact of maintenance plans on maintenance costs, asset risks and traffic network flow loss risks is calculated, further addressing the limitations of traditional single bridge assessment. Through reinforcement learning and pointer networks, the embodiments of the present invention determine the importance of bridges affecting traffic network flow and then give a bridge maintenance priority ranking. Under the constraints of limited maintenance budgets, better maintenance decisions for large-scale bridge groups can be made, and better traffic network performance can be achieved through the efficient use of maintenance resources.

[0140] Figure 5 This is a block diagram of a device for dynamic decision-making of bridge group maintenance priority based on reinforcement learning according to an exemplary embodiment. The device is used in a method for dynamic decision-making of bridge group maintenance priority based on reinforcement learning. Figure 5 The apparatus includes a first construction unit 510, a design unit 520, a second construction unit 530, and a training unit 540.

[0141] The first construction unit 510 is used to construct the topological structure of the bridge network and the road; define the state space, maintenance action space and state transition matrix of the bridge;

[0142] A design unit 520 is used to define a reliability index corresponding to the state of the bridge, and based on the topological structure of the bridge network and the road, design a comprehensive reward function based on maintenance cost, asset risk and traffic network capacity loss risk;

[0143] A second construction unit 530 is configured to construct a bridge maintenance decision problem based on the bridge state space, the maintenance action space, the state transition matrix, and the comprehensive reward function;

[0144] The training unit 540 is used to describe the bridge maintenance decision problem as a Markov decision process, establish a pointer network policy model using a pointer network, train the pointer network policy model using an actor-critic algorithm, and establish a maintenance decision model based on reinforcement learning; the maintenance decision model based on reinforcement learning is trained until convergence, and outputs a bridge maintenance action sequence under limited constraints.

[0145] Optionally, the state space of the bridge is expressed by the following formula (1):

[0146] (1)

[0147] Where S represents the state space of the bridge;

[0148] The maintenance action space is expressed by the following formula (2):

[0149] (2)

[0150] Among them, A represents the maintenance action space;

[0151] The state transfer matrix is ​​expressed by the following formula (3):

[0152] (3)

[0153] Where P represents the state transfer matrix.

[0154] Optionally, the comprehensive reward function based on maintenance cost, asset risk and traffic network capacity loss risk is expressed by the following formula (4):

[0155] (4)

[0156] in, The asset risk corresponding to the bridge status is expressed by the following formula (5):

[0157] (5)

[0158] in, The risk of traffic capacity loss of the bridge network corresponding to the bridge state is expressed by the following formula (6):

[0159] (6)

[0160] in, represents the comprehensive reward function; represents direct maintenance cost; represents the direct maintenance cost and asset risk scaling factor; represents the risk weight coefficient of bridge network traffic capacity loss; represents the direct maintenance cost and asset risk scaling factor; represents the failure probability corresponding to the bridge state; represents the cost of asset failure; Indicates the bridge failure combination corresponding to the bridge status; Represents the traffic capacity loss of the bridge network under the bridge failure combination.

[0161] Optionally, the training unit 540 is configured to:

[0162] A pointer network is used to construct a pointer policy network model. Bridge attributes, bridge status, and maintenance actions are input into an encoder to generate feature codes. A decoder integrates the feature codes, current bridge information, and the numbers of traversed bridges. An attention mechanism is used to generate the element with the largest weight as the next bridge to be repaired and the bridge maintenance action. Based on the bridges and bridge maintenance actions, a preliminary maintenance cost is calculated.

[0163] Based on the pointer policy network model, an actor-critic network is constructed. The actor-critic network includes a pointer policy network and a value network. The pointer policy network is used to output the bridge maintenance priority and maintenance action strategy based on the bridge status. The value network is used to predict the state value based on the bridge status and the information processed by the encoder.

[0164] Construct an advantage function; based on the advantage function, construct the pointer strategy network loss function and the value network loss function;

[0165] Repairs are performed according to the bridge maintenance priority ranking generated by the pointer policy network to obtain the actual reward value; the mean square error between the actual reward value and the estimated reward value generated by the value network is used as the optimization target, and the actor-critic network is trained using the gradient descent method until convergence through the pointer policy network loss function and the value network loss function, outputting the ranking of bridge maintenance actions under limited constraints.

[0166] Optionally, the advantage function is expressed by the following formula (7):

[0167] (7)

[0168] Among them, r represents the reward obtained at the current time step; represents the discount factor; Represents the value network's valuation of the next state; Represents the value network's valuation of the current state; is the advantage function, which represents the benefit of performing action a in state s; s represents the current state of the bridge; a represents the maintenance action taken.

[0169] Optionally, the loss function of the pointer strategy network is expressed by the following formula (8):

[0170] (8)

[0171] in, represents the loss of the pointer strategy network; E represents the expected operation; Represents the policy function About its implicit parameters The logarithmic gradient of ; where It represents the probability of taking action a in state s.

[0172] Optionally, the loss function of the value network is expressed by the following formula (9):

[0173] (9)

[0174] in, Represents the loss of the value network.

[0175] The embodiment of the present invention first constructs the topological structure of the bridge network and the road; defines the state space, maintenance action space and state transition matrix of the bridge; defines the reliability index corresponding to the state of the bridge, and designs a comprehensive reward function based on maintenance cost, asset risk and traffic network capacity loss risk; secondly, constructs a bridge maintenance decision problem based on the state space, maintenance action space, state transition matrix and comprehensive reward function of the bridge; finally, describes the bridge maintenance decision problem as a Markov decision process, uses a pointer network to establish a pointer network policy model, uses the actor-critic algorithm to train the pointer network policy model, and establishes a maintenance decision model based on reinforcement learning; trains the maintenance decision model based on reinforcement learning until convergence, and outputs a ranking of bridge maintenance actions under limited constraints.

[0176] By constructing a maintenance decision model based on reinforcement learning, the embodiments of the present invention can accurately simulate the degradation of bridge status and provide reliable data for subsequent decision-making. By constructing the impact law of traffic network flow loss, the impact of maintenance plans on maintenance costs, asset risks and traffic network flow loss risks is calculated, further addressing the limitations of traditional single bridge assessment. Through reinforcement learning and pointer networks, the embodiments of the present invention determine the importance of bridges affecting traffic network flow and then give a bridge maintenance priority ranking. Under the constraints of limited maintenance budgets, better maintenance decisions for large-scale bridge groups can be made, and better traffic network performance can be achieved through the efficient use of maintenance resources.

[0177] Figure 6 is a structural diagram of a bridge group maintenance priority dynamic decision-making device based on reinforcement learning provided by an embodiment of the present invention, such as Figure 6 As shown, the bridge group maintenance priority dynamic decision-making device based on reinforcement learning may include the above Figure 5 Optionally, the bridge group maintenance priority dynamic decision-making device 610 based on reinforcement learning may include a first processor 2001 .

[0178] Optionally, the bridge group maintenance priority dynamic decision-making device 610 based on reinforcement learning may further include a memory 2002 and a transceiver 2003 .

[0179] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0180] The following combination Figure 6 The components of the bridge group maintenance priority dynamic decision-making device 610 based on reinforcement learning are specifically introduced:

[0181] The first processor 2001 is the control center of the reinforcement learning-based bridge group maintenance priority dynamic decision-making device 610, and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), or application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs) or one or more field programmable gate arrays (FPGAs).

[0182] Optionally, the first processor 2001 can execute various functions of the bridge group maintenance priority dynamic decision-making device 610 based on reinforcement learning by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0183] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 6 CPU0 and CPU1 are shown in FIG.

[0184] In a specific implementation, as an embodiment, the bridge group maintenance priority dynamic decision-making device 610 based on reinforcement learning may also include multiple processors, such as Figure 6 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0185] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0186] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and accessed through the interface circuit ( Figure 6 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0187] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0188] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 6 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0189] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and be connected to the bridge group maintenance priority dynamic decision-making device 610 based on reinforcement learning through an interface circuit ( Figure 6 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0190] It should be noted that Figure 6 The structure of the reinforcement learning-based bridge group maintenance priority dynamic decision-making device 610 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0191] In addition, the technical effects of the bridge group maintenance priority dynamic decision-making device 610 based on reinforcement learning can refer to the technical effects of the bridge group maintenance priority dynamic decision-making method based on reinforcement learning described in the above method embodiment, and will not be repeated here.

[0192] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0193] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0194] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0195] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0196] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0197] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0198] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0199] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0200] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0201] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0202] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0203] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0204] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A dynamic decision-making method for bridge group maintenance priority based on reinforcement learning, characterized by: The method comprises: S1. Construct the topological structure of the bridge network and roads; define the bridge state space, maintenance action space, and state transition matrix; S2. Define the reliability index corresponding to the bridge status. Based on the topological structure of the bridge network and roads, design a comprehensive reward function based on maintenance cost, asset risk, and transportation network capacity loss risk. S3. Constructing a bridge maintenance decision problem based on the bridge state space, maintenance action space, state transition matrix, and the comprehensive reward function; S4. Describe the bridge maintenance decision-making problem as a Markov decision process, use a pointer network to establish a pointer network policy model, and use the actor-critic algorithm to train the pointer network policy model to obtain a maintenance decision model based on reinforcement learning. The maintenance decision model based on reinforcement learning is trained until convergence, and the output is the ranking of bridge maintenance actions under limited constraints.

2. The dynamic decision-making method for bridge group maintenance priority based on reinforcement learning according to claim 1 is characterized in that: The state space of the bridge is expressed by the following formula (1): （1） Where S represents the state space of the bridge; The maintenance action space is expressed by the following formula (2): （2） Among them, A represents the maintenance action space; The state transfer matrix is ​​expressed by the following formula (3): （3） Where P represents the state transfer matrix.

3. The dynamic decision-making method for bridge group maintenance priority based on reinforcement learning according to claim 1 is characterized in that: The comprehensive reward function based on maintenance cost, asset risk and transportation network capacity loss risk is expressed by the following formula (4): （4） in, The asset risk corresponding to the bridge status is expressed by the following formula (5): （5） in, The risk of traffic capacity loss of the bridge network corresponding to the bridge state is expressed by the following formula (6): （6） in, represents the comprehensive reward function; represents direct maintenance cost; represents the direct maintenance cost and asset risk scaling factor; represents the risk weight coefficient of bridge network traffic capacity loss; represents the direct maintenance cost and asset risk scaling factor; represents the failure probability corresponding to the bridge state; represents the cost of asset failure; Indicates the bridge failure combination corresponding to the bridge status; Represents the traffic capacity loss of the bridge network under the bridge failure combination.

4. The method for dynamic decision-making of bridge group maintenance priority based on reinforcement learning according to claim 1 is characterized in that: S4 describes the bridge maintenance decision problem as a Markov decision process, uses a pointer network to establish a pointer network strategy model, uses an Actor-Critic algorithm to train the pointer network strategy model, and establishes a maintenance decision model based on reinforcement learning; The reinforcement learning-based maintenance decision model is trained until convergence, and the output is a ranking of bridge maintenance actions under limited constraints, including: S41. Use a pointer network to construct a pointer strategy network model. Input bridge attributes, bridge status, and maintenance actions into an encoder to generate feature codes. Use a decoder to integrate the feature codes, current bridge information, and the numbers of traversed bridges. Use an attention mechanism to generate the bridge and maintenance action with the largest weight as the next bridge to be repaired and the bridge maintenance action. Calculate preliminary maintenance costs based on the bridges and bridge maintenance actions. S42. Construct an Actor-Critic network based on the pointer policy network model; wherein the Actor-Critic network includes a pointer policy network and a value network; wherein the pointer policy network is used to output a bridge maintenance priority and a maintenance action strategy based on the bridge status; wherein the value network is used to predict the state value based on the bridge status and the information processed by the encoder; S43. Construct an advantage function; based on the advantage function, construct a pointer strategy network loss function and a value network loss function; S44. Perform repairs according to the bridge repair priority ranking generated by the pointer policy network to obtain the actual reward value; take the mean square error between the actual reward value and the estimated reward value generated by the value network as the optimization target, and use the gradient descent method to train the Actor-Critic network through the pointer policy network loss function and the value network loss function until convergence, and output the bridge repair action ranking under limited constraints.

5. The method for dynamic decision-making of bridge group maintenance priority based on reinforcement learning according to claim 4 is characterized in that: The advantage function is expressed by the following formula (7): （7） Among them, r represents the reward obtained at the current time step; represents the discount factor; Represents the value network's valuation of the next state; Represents the value network's valuation of the current state; is the advantage function, which represents the benefit of performing action a in state s; s represents the current state of the bridge; a represents the maintenance action taken.

6. The method for dynamic decision-making of bridge group maintenance priority based on reinforcement learning according to claim 4 is characterized in that: The loss function of the pointer strategy network is expressed by the following formula (8): （8） in, represents the loss of the pointer strategy network; E represents the expected operation; Represents the policy function About its implicit parameters The logarithmic gradient of ; where It represents the probability of taking action a in state s.

7. The method for dynamic decision-making of bridge group maintenance priority based on reinforcement learning according to claim 4 is characterized in that: The loss function of the value network is expressed by the following formula (9): （9） in, Represents the loss of the value network.

8. A device for dynamically deciding bridge group maintenance priorities based on reinforcement learning, wherein the device is used to implement the method for dynamically deciding bridge group maintenance priorities based on reinforcement learning as claimed in any one of claims 1 to 7, characterized in that: The device comprises: The first construction unit is used to construct the topological structure of the bridge network and the road; define the state space, maintenance action space and state transition matrix of the bridge; A design unit is used to define the reliability index corresponding to the bridge status. Based on the topological structure of the bridge network and the road, a comprehensive reward function is designed based on the maintenance cost, asset risk and the risk of loss of transportation network capacity; The second construction unit is used to construct a bridge maintenance decision problem based on the bridge state space, the maintenance action space, the state transition matrix and the comprehensive reward function; The training unit is used to describe the bridge maintenance decision-making problem as a Markov decision process, establish a pointer network policy model using a pointer network, train the pointer network policy model using an actor-critic algorithm, and establish a maintenance decision model based on reinforcement learning; the maintenance decision model based on reinforcement learning is trained until convergence, and the bridge maintenance action ranking under limited constraints is output.

9. A dynamic decision-making device for bridge group maintenance priority based on reinforcement learning, characterized in that: The bridge group maintenance priority dynamic decision-making device based on reinforcement learning includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images