Bridge group maintenance priority dynamic decision method and device based on reinforcement learning

Abstract

The application provides a kind of bridge group maintenance priority dynamic decision method and device based on reinforcement learning, it is related to bridge wisdom maintenance technical field.The method comprises: the topological structure of bridge network and road is constructed;Define the state space of bridge, maintenance action space and state transition matrix;Define the reliability index corresponding to the state of bridge, based on topological structure, design comprehensive reward function based on maintenance cost, asset risk and traffic network capacity loss risk;Bridge maintenance decision problem is constructed;Bridge maintenance decision problem is described as Markov decision process, pointer network is used to establish pointer network strategy model, Actor-Critic algorithm is used to train pointer network strategy model, and reinforcement learning-based maintenance decision model is obtained;The reinforcement learning-based maintenance decision model is trained until convergence, and the bridge maintenance action sequence under the limited constraint is output.The limitation problem of traditional single bridge evaluation can be solved by using the application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of bridge intelligent maintenance, in particular to a bridge group maintenance priority dynamic decision-making method and device based on reinforcement learning. BACKGROUND

[0002] During the service process, bridges are subjected to the effects of environment and load, resulting in the degradation of bridge state. As an important part of infrastructure network, bridge maintenance is affected by factors such as funds, environment and social needs. The existing bridge management system mainly optimizes budget allocation and formulates maintenance strategies, but there is a bottleneck problem of optimization efficiency.

[0003] At present, the management of bridge groups in China mainly adopts a periodic maintenance strategy or a state-based maintenance strategy. The periodic maintenance strategy adopts a predetermined fixed maintenance period, which cannot flexibly adapt to maintenance needs. The state-based maintenance strategy adopts a fixed maintenance strategy according to the bridge state, but is subject to budget constraints, resulting in the inability to timely maintain all bridges, and the maintenance budget cannot be reasonably allocated to important bridges.

[0004] The existing bridge maintenance decision-making method based on dynamic programming and genetic algorithm is a static optimization method, which has low intelligence, lacks consideration of regional bridge group maintenance decision-making problems, and is difficult to handle large-scale state space and action space and long-term non-stationary environment decision-making problems.

[0005] The existing bridge maintenance priority ranking is based on maintenance cost or decision-maker subjective experience for priority ranking, which is difficult to ensure optimization of decision-making. At the same time, the state of the bridge affects the service performance of the traffic network, and there are differences in the evolution law of different bridges, resulting in changes in the maintenance priority of the bridge. The existing research has not fully considered the influence of the state evolution of the bridge group on the traffic network flow, and it is difficult to realize the dynamic maintenance priority ranking of the network capacity demand change and bridge maintenance under the limitation of budget maintenance funds. SUMMARY

[0006] In order to solve the technical problems of low intelligence, lack of consideration of regional bridge group maintenance decision-making problems, difficulty in handling large-scale state space and action space and long-term non-stationary environment decision-making problems, and difficulty in realizing the dynamic maintenance priority ranking of network capacity demand change and bridge maintenance under the limitation of budget maintenance funds, the embodiments of the present application provide a bridge group maintenance priority dynamic decision-making method and device based on reinforcement learning. The technical solution is as follows:

[0007] On the one hand, a bridge group maintenance priority dynamic decision-making method based on reinforcement learning is provided, which is realized by a bridge group maintenance priority dynamic decision-making device based on reinforcement learning. The method comprises:

[0008] S1, construct a topology of a bridge network and a road; define a state space of a bridge, a maintenance action space, and a state transition matrix;

[0009] S2, define a reliability index corresponding to a state of a bridge, and design a comprehensive reward function based on maintenance cost, asset risk, and traffic network capacity loss risk based on a topology of a bridge network and a road;

[0010] S3, construct a bridge maintenance decision problem according to a state space of a bridge, a maintenance action space, a state transition matrix, and the comprehensive reward function;

[0011] S4, describe the bridge maintenance decision problem as a Markov decision process, establish a pointer network policy model using a pointer network, train the pointer network policy model using an Actor-Critic algorithm, and obtain a maintenance decision model based on reinforcement learning; train the maintenance decision model based on reinforcement learning until convergence, and output a bridge maintenance action sequence under limited constraints.

[0012] Optionally, the state space of the bridge is represented by the following formula (1):

[0013]

[0014] wherein S represents the state space of the bridge;

[0015] wherein the maintenance action space is represented by the following formula (2):

[0016]

[0017] wherein A represents the maintenance action space;

[0018] wherein the state transition matrix is represented by the following formula (3):

[0019]

[0020] wherein P represents the state transition matrix.

[0021] Optionally, the comprehensive reward function based on maintenance cost, asset risk, and traffic network capacity loss risk is represented by the following formula (4):

[0022] Reward=(C AST +R ECON ) / λ NET +w NET R NET / λ AST (4)

[0023] wherein R ASTrepresents the asset risk corresponding to the bridge state, which is represented by the following formula (5):

[0024] R AST =∑p f (s)C f (5)

[0025] wherein, R NET represents the bridge network traffic capacity loss risk corresponding to the bridge state, which is represented by the following formula (6):

[0026] R NET =∑p(c|s)[F0-F(c)] (6)

[0027] wherein, Reward represents a comprehensive reward function; C M represents a direct maintenance cost; λ ECON represents a direct maintenance cost and asset risk scaling coefficient; w NET represents a bridge network traffic capacity loss risk weight coefficient; λ NET represents a direct maintenance cost and asset risk scaling coefficient; p f (s) represents a failure probability corresponding to the bridge state; C f represents an asset failure cost; p(c|s) represents a bridge failure combination corresponding to the bridge state; [F0-F(c)] represents a bridge network traffic capacity loss under the bridge failure combination.

[0028] Optionally, the S4 describes the bridge maintenance decision problem as a Markov decision process, adopts a pointer network to establish a pointer network policy model, adopts an Actor-Critic algorithm to train the pointer network policy model, and establishes a maintenance decision model based on reinforcement learning; the maintenance decision model based on reinforcement learning is trained until convergence, and a bridge maintenance action sequence under limited constraints is output, including:

[0029] S41, a pointer network is adopted to construct a pointer policy network model, and bridge attributes, bridge states and maintenance actions are input into an encoder to generate feature codes; a decoder is adopted to integrate the feature codes, current bridge information and traversed part of bridge numbers, and an element with the largest weight is generated through an attention mechanism as a bridge to be maintained and a bridge maintenance action; a preliminary maintenance cost is calculated according to the bridge and the bridge maintenance action;

[0030] S42, an Actor-Critic network is constructed according to the pointer policy network model; wherein the Actor-Critic network includes a pointer policy network and a value network; wherein the pointer policy network is used to output a bridge maintenance priority and a maintenance action policy according to a bridge state; and wherein the value network is used to perform state value prediction according to the bridge state and information processed by the encoder.

[0031] S43, constructing an advantage function; according to the advantage function, constructing a pointer strategy network loss function and a value network loss function;

[0032] S44, repairing according to the bridge maintenance priority ranking generated by the pointer strategy network to obtain an actual reward value; taking a mean square error of the actual reward value and an estimated reward value generated by the value network as an optimization target, training the Actor-Critic network by using a gradient descent method until convergence through the pointer strategy network loss function and the value network loss function, and outputting a bridge maintenance action ranking under a limited constraint.

[0033] Optionally, the advantage function is represented by the following formula (7):

[0034] Adv(s, a) = r + γV(s') - V(s) (7)

[0035] wherein r represents a reward obtained at a current time step; γ represents a discount factor; V(s') represents an estimated value of a next state by the value network; V(s) represents an estimated value of a current state by the value network; Adv(s, a) is an advantage function, representing a return of performing an action a under a state s; s represents a current state of the bridge; and a represents a maintenance action taken.

[0036] Optionally, the loss function of the pointer strategy network is represented by the following formula (8):

[0037]

[0038] wherein ActorLoss represents a loss of the pointer strategy network; E represents an expectation operation; represents a logarithmic gradient of the policy function π(a|s) with respect to its implicit parameters θ; wherein π(a|s) represents a probability of taking an action a under a state s.

[0039] Optionally, the loss function of the value network is represented by the following formula (9):

[0040] CriticLoss = E[Adv(s, a) 2 ] (9)

[0041] wherein CriticLoss represents a loss of the value network.

[0042] On the other hand, a bridge group maintenance priority dynamic decision device based on reinforcement learning is provided, which is applied to a bridge group maintenance priority dynamic decision method based on reinforcement learning, and the device comprises:

[0043] The first building unit is used to construct the topology of the bridge network and roads; it defines the state space, maintenance action space, and state transition matrix of the bridge.

[0044] The design unit is used to define the reliability index corresponding to the state of the bridge, and to design a comprehensive reward function based on maintenance cost, asset risk and traffic network capacity loss risk, based on the topology of the bridge network and road.

[0045] The second construction unit is used to construct the bridge maintenance decision problem based on the bridge's state space, maintenance action space, state transition matrix, and the comprehensive reward function.

[0046] The training unit describes the bridge maintenance decision problem as a Markov decision process. It uses a pointer network to build a pointer network policy model, and uses the Actor-Critic algorithm to train the pointer network policy model to build a reinforcement learning-based maintenance decision model. The reinforcement learning-based maintenance decision model is trained until convergence, and the output is a ranking of bridge maintenance actions under finite constraints.

[0047] Optionally, the state space of the bridge is represented by the following formula (1):

[0048]

[0049] Where S represents the state space of the bridge;

[0050] The maintenance action space is represented by the following formula (2):

[0051]

[0052] Where A represents the maintenance action space;

[0053] The state transition matrix is ​​represented by the following formula (3):

[0054]

[0055] Where P represents the state transition matrix.

[0056] Optionally, the comprehensive reward function based on maintenance costs, asset risk, and traffic network capacity loss risk is expressed by the following formula (4):

[0057] Reward = (C M +R AST ) / λ ECON +w NET R NET / λ NET (4)

[0058] Among them, RAST represents the asset risk corresponding to the bridge state, which is represented by the following formula (5):

[0059] R AST =∑p f (s)C f (5)

[0060] wherein, R ENT represents the bridge network traffic capacity loss risk corresponding to the bridge state, which is represented by the following formula (6):

[0061] R NET =∑p(c|s)[F0-F(c)] (6)

[0062] wherein, Reward represents a comprehensive reward function; C M represents a direct maintenance cost; λ ECON represents a direct maintenance cost and asset risk scaling coefficient; w NET represents a bridge network traffic capacity loss risk weight coefficient; λ NET represents a direct maintenance cost and asset risk scaling coefficient; p f (s) represents a failure probability corresponding to the bridge state; C f represents an asset failure cost; p(c|s) represents a bridge failure combination corresponding to the bridge state; [F0-F(c)] represents a bridge network traffic capacity loss under the bridge failure combination.

[0063] Optionally, the training unit is configured to:

[0064] The pointer network is used to construct a pointer policy network model, and the bridge attribute, the bridge state and the maintenance action are input into an encoder to generate feature encoding. A decoder integrates the feature encoding, the current bridge information and the traversed part of the bridge number, generates the element with the maximum weight as the next maintenance bridge and the bridge maintenance action through an attention mechanism, and calculates a preliminary maintenance cost according to the bridge and the bridge maintenance action.

[0065] An Actor-Critic network is constructed according to the pointer policy network model, wherein the Actor-Critic network comprises a pointer policy network and a value network, wherein the pointer policy network is configured to output a bridge maintenance priority and a maintenance action policy according to the bridge state, and the value network is configured to perform state value prediction according to the bridge state and information processed by the encoder.

[0066] An advantage function is constructed, and a pointer policy network loss function and a value network loss function are constructed according to the advantage function.

[0067] The bridge maintenance priority ranking generated by the pointer policy network is repaired to obtain an actual reward value; a mean square error of the actual reward value and an estimated reward value generated by the value network is taken as an optimization target, the Actor-Critic network is trained by using a gradient descent method until convergence is achieved through a loss function of the pointer policy network and a loss function of the value network, and a bridge maintenance action ranking under a limited constraint is output.

[0068] Optionally, the advantage function is represented by the following formula (7):

[0069] Adv(s, a) = r + γV(s') - V(s) (7)

[0070] wherein r represents a reward obtained at a current time step; γ represents a discount factor; V(s') represents an estimated value of the value network for a next state; V(s) represents an estimated value of the value network for a current state; Adv(s, a) is an advantage function, representing a return of performing an action a in a state s; s represents a current state of the bridge; and a represents a maintenance action taken.

[0071] Optionally, the loss function of the pointer policy network is represented by the following formula (8):

[0072]

[0073] wherein ActorLoss represents a loss of the pointer policy network; E represents an expectation operation; represents a logarithmic gradient of the policy function π(a|s) with respect to its implicit parameter θ; wherein π(a|s) represents a probability of taking an action a in a state s.

[0074] Optionally, the loss function of the value network is represented by the following formula (9):

[0075] CriticLoss = E[Adv(s, a) 2 ] (9)

[0076] wherein CriticLoss represents a loss of the value network.

[0077] In another aspect, a bridge group maintenance priority dynamic decision device based on reinforcement learning is provided, which comprises a processor and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement any one of the above bridge group maintenance priority dynamic decision methods based on reinforcement learning.

[0078] In another aspect, a computer-readable storage medium is provided, the storage medium storing at least one instruction, the at least one instruction being loaded and executed by a processor to implement any one of the above bridge group maintenance priority dynamic decision-making methods based on reinforcement learning.

[0079] The technical scheme provided by the embodiment of the application has at least the following beneficial effects:

[0080] The embodiment of the application first constructs the topological structure of the bridge network and the road; defines the state space, the maintenance action space and the state transition matrix of the bridge; defines the reliability index corresponding to the state of the bridge, and designs a comprehensive reward function based on the maintenance cost, asset risk and traffic network capacity loss risk; secondly, according to the state space, the maintenance action space, the state transition matrix and the comprehensive reward function of the bridge, a bridge maintenance decision-making problem is constructed; finally, the bridge maintenance decision-making problem is described as a Markov decision process, a pointer network strategy model is established by using the pointer network, the Actor-Critic algorithm is used to train the pointer network strategy model, and a maintenance decision-making model based on reinforcement learning is established; the maintenance decision-making model based on reinforcement learning is trained until convergence, and the bridge maintenance action sequence under the limited constraint is output.

[0081] The embodiment of the application can accurately simulate the bridge degradation state by constructing the maintenance decision-making model based on reinforcement learning, and provide reliable data for subsequent decision-making; by constructing the topological structure of the bridge network and the road, calculating the influence law of the bridge state on the traffic network flow loss risk, and calculating the influence of the maintenance scheme on the maintenance cost, asset risk and traffic network flow loss risk, the limitation of traditional single bridge evaluation is further solved; the embodiment of the application judges the important bridge affecting the traffic network flow by using reinforcement learning and the pointer network, and then gives the bridge maintenance priority sequence, so that the large-scale bridge group maintenance decision-making can be better performed under the limited maintenance budget constraint, and the traffic network nature is improved by efficient use of maintenance resources. BRIEF DESCRIPTION OF DRAWINGS

[0082] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0083] Figure 1 is a flow chart of a bridge group maintenance priority dynamic decision-making method based on reinforcement learning provided by the embodiment of the application;

[0084] Figure 2 is a topological structure diagram of a bridge network provided by the embodiment of the application;

[0085] Figure 3 is a maintenance priority ranking schematic diagram based on reinforcement learning provided by an embodiment of the application;

[0086] Figure 4 is a bridge natural degradation state distribution diagram provided by an embodiment of the application;

[0087] Figure 5 is a bridge group maintenance priority dynamic decision device block diagram based on reinforcement learning provided by an embodiment of the application;

[0088] Figure 6 is a structure schematic diagram of a bridge group maintenance priority dynamic decision device based on reinforcement learning provided by an embodiment of the application. DETAILED DESCRIPTION

[0089] The technical solutions in the application will be described below with reference to the drawings.

[0090] In the embodiments of the application, the words such as "example", "for example" and the like are used to represent an example, illustration or description. Any embodiment or design scheme described as "example" in the application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific manner. In addition, in the embodiments of the application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0091] In the embodiments of the application, "image" and "picture" can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized. "Of", "corresponding" and "corresponding" can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized.

[0092] In the embodiments of the application, sometimes the subscript such as W1 can be written in the form of non-subscript such as W1, and the meanings expressed are consistent when the distinction is not emphasized.

[0093] To make the technical problems, technical solutions and advantages to be solved by the application more clear, the following will be described in detail with reference to the drawings and specific embodiments.

[0094] The embodiment of the application provides a bridge group maintenance priority dynamic decision method based on reinforcement learning, which can be realized by a bridge group maintenance priority dynamic decision device based on reinforcement learning. The bridge group maintenance priority dynamic decision device based on reinforcement learning can be a terminal or a server. For example, Figure 1The flow chart of the bridge group maintenance priority dynamic decision method based on reinforcement learning shown in the figure, the processing flow of the method can include the following steps:

[0095] S1, construct the topology of the bridge network and the road; define the state space of the bridge, the maintenance action space and the state transition matrix.

[0096] In a feasible implementation manner, as Figure 2 The topological structure diagram of the bridge network provided by the embodiment of the application is shown; wherein the bridge network is abstracted into a topological result form of a directed graph, which can be represented as G=(N, E); wherein G represents a directed graph, each node N in the directed graph represents a road intersection; the node information includes the intersection position coordinates; each edge E of the directed graph represents the connecting section between the intersections; the edge information includes the OD traffic flow and the bridge information.

[0097] Optionally, the state space of the bridge is represented by the following formula (1):

[0098]

[0099] Wherein S represents the state space of the bridge;

[0100] Wherein the maintenance action space is represented by the following formula (2):

[0101]

[0102] Wherein A represents the maintenance action space;

[0103] Wherein the state transition matrix is represented by the following formula (3):

[0104]

[0105] Wherein P represents the state transition matrix.

[0106] Wherein the state transition matrix includes five state transition matrices corresponding to different maintenance actions, and the dimension of each action state transition matrix is d s ×d a If the state of the bridge at time t is represented as s t , after taking any maintenance action a i , the state of the bridge is sampled based on the state transition matrix p i , and becomes s t+1 .

[0107] S2, define the reliability index corresponding to the state of the bridge, design a comprehensive reward function based on the maintenance cost, asset risk and traffic network capacity loss risk based on the topology of the bridge network and the road.

[0108] Optionally, the comprehensive reward function based on the maintenance cost, asset risk and traffic network capacity loss risk is represented by the following formula (4):

[0109] Reward = (C M + R AST ) / λ ECON + w NET R NET / λ NET (4)

[0110] wherein R AST represents the asset risk corresponding to the bridge state, and is represented by the following formula (5):

[0111] R AST =∑p f (s)C f (5)

[0112] wherein R NET represents the bridge network traffic capacity loss risk corresponding to the bridge state, and is represented by the following formula (6):

[0113] R NET =∑p(c|s)[F0-F(c)] (6)

[0114] wherein Reward represents the comprehensive reward function; C M represents the direct maintenance cost; λ ECON represents the direct maintenance cost and asset risk scaling coefficient; w NET represents the bridge network traffic capacity loss risk weight coefficient; λ NET represents the direct maintenance cost and asset risk scaling coefficient; p f (s) represents the failure probability corresponding to the bridge state; C f represents the asset failure cost; p(c|s) represents the bridge failure combination corresponding to the bridge state; [F0-F(c)] represents the bridge network traffic capacity loss under the bridge failure combination.

[0115] S3, according to the state space of the bridge, the maintenance action space, the state transition matrix and the comprehensive reward function, a bridge maintenance decision problem is constructed.

[0116] wherein the Markov decision process includes a five-tuple which can be represented as: (S, A, P, R, γ); wherein S represents ; A represents ; P represents ; R represents the comprehensive reward function; γ represents the discount factor, and the value is [0, 1], which is used to balance the relationship between the current return and the future return, and the lower the value is, the more attention will be paid to the current reward, focusing on short-term cost saving, and vice versa emphasizing long-term income.

[0117] S4, describe the bridge maintenance decision problem as a Markov decision process, establish a pointer network policy model using a pointer network, train the pointer network policy model using an Actor-Critic algorithm, and obtain a maintenance decision model based on reinforcement learning; train the maintenance decision model based on reinforcement learning until convergence, and output the bridge maintenance action sequence under the limited constraint.

[0118] Wherein, the pointer network is a sequence-to-sequence learning model, which can solve the problem that the traditional seq2seq model cannot handle through the attention mechanism, that is, the output sequence changes with the change of the input sequence length.

[0119] Wherein, as Figure 3 shown is a maintenance priority ranking diagram based on reinforcement learning provided by the embodiment of the application; in a feasible implementation, the bridge attributes, the current state of the bridge and the maintenance actions that can be taken are input into the encoder to generate feature encoding, the feature encoding is input into the decoder, and the element with the maximum weight is generated as the bridge to be maintained and the maintenance action of the bridge through the attention mechanism; the preliminary maintenance cost is calculated according to the current maintenance bridge and the maintenance action of the bridge; when the maintenance budget is sufficient, the maintenance is agreed and the maintenance action is completed; when the maintenance budget is insufficient, the maintenance is refused.

[0120] Optionally, the specific implementation process of S4 includes S41-S414:

[0121] S41, a pointer strategy network model is constructed using a pointer network, and the bridge attributes, the current state of the bridge and the maintenance actions that can be taken are input into the encoder to generate feature encoding; the decoder integrates the feature encoding, the current bridge information and the traversed part of the bridge number, and generates the bridge with the maximum weight and the corresponding selected action as the bridge to be maintained and the maintenance action of the bridge through the attention mechanism in the next step; the preliminary maintenance cost is calculated according to the single-bridge maintenance bridge and the maintenance action of the bridge;

[0122] Wherein, in each step, the bridge with the maximum weight is taken as the final output according to the attention mechanism.

[0123] Wherein, when the maintenance budget is sufficient, the maintenance action is completed; when the maintenance budget is insufficient, the maintenance action is refused and the maintenance is not taken.

[0124] S42, construct an Actor-Critic network according to the pointer strategy network model; wherein the Actor-Critic network includes: a pointer strategy network and a value network; wherein the pointer strategy network is used to output the bridge maintenance priority and the maintenance action strategy according to the bridge state; wherein the value network is used to predict the state value according to the bridge state and the information processed by the encoder;

[0125] S43, construct an advantage function; according to the advantage function, construct a pointer strategy network loss function and a value network loss function;

[0126] Optionally, the advantage function is represented by the following formula (7):

[0127] Adv (s, a) = r + γV (s') - V (s) (7)

[0128] Wherein, r represents the reward obtained at the current time step; γ represents a discount factor; V (s') represents the estimated value of the next state by the value network; V (s) represents the estimated value of the current state by the value network; Adv (s, a) is the advantage function, which represents the return of performing action a in state s; s represents the current state of the bridge; a represents the maintenance action taken.

[0129] Optionally, the loss function of the pointer strategy network is represented by the following formula (8):

[0130]

[0131] Wherein, ActorLoss represents the loss of the pointer strategy network; E represents an expectation operation; Represents the logarithmic gradient of the policy function π (a | s) with respect to its implicit parameters θ; wherein, π (a | s) represents the probability of taking action a in state s.

[0132] Optionally, the loss function of the value network is represented by the following formula (9):

[0133] CriticLoss = E [Adv (s, a) 2 ] (9)

[0134] Wherein, CriticLoss represents the loss of the value network.

[0135] S44, according to the bridge maintenance priority ranking generated by the pointer strategy network, perform maintenance, obtain an actual reward value; take the mean square error of the actual reward value and the estimated reward value generated by the value network as an optimization target, train the Actor-Critic network by using the gradient descent method until convergence through the pointer strategy network loss function and the value network loss function, and output the bridge maintenance action ranking under the limited constraint.

[0136] Wherein, by using the embodiment of the present application, a maintenance strategy with lower bridge group maintenance cost and lower risk of traffic network capacity loss can be realized.

[0137] Wherein, as Figure 4A bridge natural degradation state distribution diagram provided by the embodiment of the application is shown, wherein the state degree is divided into state 0, state 1, state 2, state 3 and state 4.

[0138] The embodiment of the application firstly constructs the topological structure of the bridge network and the road, defines the state space, the maintenance action space and the state transition matrix of the bridge, defines the reliability index corresponding to the state of the bridge, and designs a comprehensive reward function based on the maintenance cost, the asset risk and the traffic network capacity loss risk; secondly, according to the state space, the maintenance action space, the state transition matrix and the comprehensive reward function of the bridge, a bridge maintenance decision problem is constructed; finally, the bridge maintenance decision problem is described as a Markov decision process, a pointer network strategy model is established by using the pointer network, the pointer network strategy model is trained by using the Actor-Critic algorithm, and a maintenance decision model based on reinforcement learning is established; the maintenance decision model based on reinforcement learning is trained until convergence, and the bridge maintenance action sequence under the limited constraint is output.

[0139] The embodiment of the application can accurately simulate the bridge state degradation by constructing the maintenance decision model based on reinforcement learning, and provide reliable data for subsequent decision-making; the limitation of traditional single bridge evaluation is further solved by constructing the traffic network flow loss influence law, calculating the influence of the maintenance scheme on the maintenance cost, the asset risk and the traffic network flow loss risk; the embodiment of the application judges the important bridge affecting the traffic network flow by reinforcement learning and the pointer network, and then gives the bridge maintenance priority sequence, so that the large-scale bridge group maintenance decision can be better made under the limited maintenance budget constraint, and the traffic network performance is better by efficient use of maintenance resources.

[0140] Figure 5 It is a bridge group maintenance priority dynamic decision device block diagram based on reinforcement learning according to an exemplary embodiment, and the device is used for the bridge group maintenance priority dynamic decision method based on reinforcement learning. Referring to Figure 5 The device includes a first construction unit 510, a design unit 520, a second construction unit 530 and a training unit 540. Wherein:

[0141] The first construction unit 510 is used for constructing the topological structure of the bridge network and the road, defining the state space, the maintenance action space and the state transition matrix of the bridge;

[0142] The design unit 520 is used for defining the reliability index corresponding to the state of the bridge, designing a comprehensive reward function based on the maintenance cost, the asset risk and the traffic network capacity loss risk based on the topological structure of the bridge network and the road;

[0143] The second construction unit 530 is configured to construct a bridge maintenance decision problem according to a state space of the bridge, a maintenance action space, a state transition matrix and the comprehensive reward function.

[0144] The training unit 540 is configured to describe the bridge maintenance decision problem as a Markov decision process, establish a pointer network policy model by using a pointer network, train the pointer network policy model by using an Actor-Critic algorithm, and establish a maintenance decision model based on reinforcement learning; and train the maintenance decision model based on reinforcement learning until convergence, and output a bridge maintenance action sequence under a limited constraint.

[0145] Optionally, the state space of the bridge is represented by the following formula (1):

[0146]

[0147] wherein S represents the state space of the bridge.

[0148] Optionally, the maintenance action space is represented by the following formula (2):

[0149]

[0150] wherein A represents the maintenance action space.

[0151] Optionally, the state transition matrix is represented by the following formula (3):

[0152]

[0153] wherein P represents the state transition matrix.

[0154] Optionally, the comprehensive reward function based on the maintenance cost, the asset risk and the traffic network capacity loss risk is represented by the following formula (4):

[0155] Reward=(C M +R AST ) / λ ECON +w NET R NET / λ NET (4)

[0156] wherein R AST represents the asset risk corresponding to the state of the bridge, and is represented by the following formula (5):

[0157] R AST =∑p f (s)C f (5)

[0158] wherein R NETa risk of loss of traffic capacity of the bridge network corresponding to the bridge state, and is represented by the following formula (6):

[0159] R NET =∑p(c|s)[F0-F(c)] (6)

[0160] wherein Reward represents a comprehensive reward function; C M represents a direct maintenance cost; λ ECON represents a direct maintenance cost and an asset risk scaling coefficient; w NET represents a risk weight coefficient of loss of traffic capacity of the bridge network corresponding to the bridge state; λ NET represents a direct maintenance cost and an asset risk scaling coefficient; p f (s) represents a failure probability corresponding to the bridge state; C f represents an asset failure cost; p(c|s) represents a bridge failure combination corresponding to the bridge state; and [F0-F(c)] represents a loss of traffic capacity of the bridge network under the bridge failure combination.

[0161] Optionally, the training unit 540 is configured to:

[0162] The pointer network is used to construct a pointer policy network model, bridge attributes, bridge states and maintenance actions are input into an encoder to generate feature codes; a decoder is used to integrate the feature codes, current bridge information and part of the traversed bridge numbers, and a weight maximum element is generated by an attention mechanism as a bridge to be maintained and a bridge maintenance action; a preliminary maintenance cost is calculated according to the bridge and the bridge maintenance action;

[0163] An Actor-Critic network is constructed according to the pointer policy network model; the Actor-Critic network includes a pointer policy network and a value network; the pointer policy network is configured to output a bridge maintenance priority and a maintenance action policy according to a bridge state; the value network is configured to perform state value prediction according to the bridge state and information processed by the encoder;

[0164] An advantage function is constructed; a pointer policy network loss function and a value network loss function are constructed according to the advantage function;

[0165] The bridge maintenance priority ranking generated by the pointer policy network is used for maintenance to obtain an actual reward value; a mean square error of the actual reward value and an estimated reward value generated by the value network is taken as an optimization target, the Actor-Critic network is trained by using a gradient descent method until convergence through the pointer policy network loss function and the value network loss function, and a bridge maintenance action ranking under a limited constraint is output.

[0166] Optionally, the advantage function is represented by the following formula (7):

[0167] Adv(s,a)=r+γV(s')-V(s) (7)

[0168] Wherein, r represents the reward obtained at the current time step;γ represents the discount factor;V(s') represents the estimated value of the next state by the value network;V(s) represents the estimated value of the current state by the value network;Adv(s,a) is the advantage function, which represents the return of performing action a in state s;s represents the current state of the bridge;a represents the maintenance action taken.

[0169] Optionally, the loss function of the pointer policy network is represented by the following formula (8):

[0170]

[0171] Wherein, ActorLoss represents the loss of the pointer policy network;E represents the expectation operation; Represents the logarithmic gradient of the policy function π(a|s) with respect to its implicit parameters θ;Wherein, π(a|s) represents the probability of taking action a in state s.

[0172] Optionally, the loss function of the value network is represented by the following formula (9):

[0173] CriticLoss=E[Adv(s,a) 2 ] (9)

[0174] Wherein, CriticLoss represents the loss of the value network.

[0175] The embodiment of the application first constructs the topological structure of the bridge network and the road;Define the state space, maintenance action space and state transition matrix of the bridge;Define the reliability index corresponding to the state of the bridge, design a comprehensive reward function based on maintenance cost, asset risk and traffic network capacity loss risk;Secondly, according to the state space, maintenance action space, state transition matrix and comprehensive reward function of the bridge, the bridge maintenance decision problem is constructed;Finally, the bridge maintenance decision problem is described as a Markov decision process, a pointer network policy model is established by using a pointer network, an Actor-Critic algorithm is used to train the pointer network policy model, and a maintenance decision model based on reinforcement learning is established;The maintenance decision model based on reinforcement learning is trained until convergence, and the bridge maintenance action sequence under the limited constraint is output.

[0176] The embodiment of the application can accurately simulate bridge state degradation by constructing a maintenance decision model based on reinforcement learning, and provide reliable data for subsequent decision-making; the embodiment of the application can further solve the limitations of traditional single bridge evaluation by constructing traffic network flow loss influence law, and calculating the influence of the maintenance scheme on maintenance cost, asset risk and traffic network flow loss risk; the embodiment of the application can determine important bridges affecting traffic network flow by using reinforcement learning and pointer network, and then give a bridge maintenance priority ranking, so that better large-scale bridge group maintenance decision-making can be made under the constraint of limited maintenance budget, and better traffic network performance can be achieved by efficient use of maintenance resources.

[0177] Figure 6 is a structural schematic diagram of a bridge group maintenance priority dynamic decision device based on reinforcement learning provided by the embodiment of the application, as Figure 6 shown, the bridge group maintenance priority dynamic decision device based on reinforcement learning can include the bridge group maintenance priority dynamic decision device based on reinforcement learning shown in Figure 5 Optionally, the bridge group maintenance priority dynamic decision device based on reinforcement learning 610 can include a first processor 2001.

[0178] Optionally, the bridge group maintenance priority dynamic decision device based on reinforcement learning 610 can further include a memory 2002 and a transceiver 2003.

[0179] Among them, the first processor 2001 and the memory 2002 and the transceiver 2003 can be connected through a communication bus.

[0180] The various constituent components of the bridge group maintenance priority dynamic decision device based on reinforcement learning 610 will be specifically introduced below: Figure 6

[0181] Among them, the first processor 2001 is the control center of the bridge group maintenance priority dynamic decision device based on reinforcement learning 610, which can be one processor or a plurality of processing elements. For example, the first processor 2001 is one or more central processing units (CPU), which can also be an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the application, such as one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA).

[0182] ​Optionally, the first processor 2001 can execute various functions of the bridge group maintenance priority dynamic decision device 610 based on reinforcement learning by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0183] In a specific implementation, as an embodiment, the first processor 2001 can include one or more CPUs, such as CPU0 and CPU1 shown in FIG. 2. Figure 6

[0184] In a specific implementation, as an embodiment, the bridge group maintenance priority dynamic decision device 610 based on reinforcement learning can also include multiple processors, such as the first processor 2001 and the second processor 2004 shown in FIG. 2. Each of these processors can be a single-CPU or a multi-CPU. The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Figure 6

[0185] The memory 2002 is configured to store software programs for executing the scheme of the present application and is controlled by the first processor 2001 to execute. The specific implementation can refer to the above method embodiments, and details are not described herein.

[0186] Optionally, the memory 2002 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, an optical disk storage (including a compact disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory 2002 can be integrated with the first processor 2001 or exist independently and be coupled with the first processor 2001 through an interface circuit (not shown in FIG. 2) of the bridge group maintenance priority dynamic decision device 610 based on reinforcement learning. The embodiments of the present application are not limited in this regard. Figure 6 ​​​

[0187] The transceiver 2003 is configured to communicate with a network device or a terminal device.

[0188] Optionally, the transceiver 2003 can include a receiver and a transmitter (not shown in the figure). Figure 6 The receiver is configured to implement a receiving function, and the transmitter is configured to implement a transmitting function.

[0189] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or can exist independently and be coupled to the first processor 2001 through an interface circuit (not shown in the figure) of the bridge group maintenance priority dynamic decision device 610 based on reinforcement learning. Figure 6 The embodiments of the present application do not make a specific limitation.

[0190] It should be noted that, Figure 6 The structure of the bridge group maintenance priority dynamic decision device 610 based on reinforcement learning shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device can include more or fewer components than those shown in the figure, or combine certain components, or different component arrangements.

[0191] In addition, the technical effects of the bridge group maintenance priority dynamic decision device 610 based on reinforcement learning can refer to the technical effects of the bridge group maintenance priority dynamic decision method based on reinforcement learning described in the above method embodiments, which will not be repeated here.

[0192] It should be understood that the first processor 2001 in the embodiments of the present application can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0193] It should also be understood that the memory in the embodiments of the present application can be volatile or nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. The volatile memory can be random access memory (RAM) used as external cache. By way of example, and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0194] The above-described embodiments can be implemented in whole or in part by software, hardware (e.g., circuitry), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0195] It should be understood that the term "and / or" herein merely describes an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can represent three cases of A alone, A and B together, and B alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects, but can also represent an "and / or" relationship, which can be understood in the context before and after.

[0196] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of the items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0197] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0198] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0199] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0200] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0201] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0202] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0203] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0204] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A bridge group maintenance priority dynamic decision method based on reinforcement learning, characterized in that, The method comprises: S1, constructing a topological structure of a bridge network and a road; defining a state space of a bridge, a maintenance action space, and a state transition matrix; S2, defining a reliability index corresponding to the state of the bridge, and designing a comprehensive reward function based on maintenance cost, asset risk, and traffic network capacity loss risk based on the topological structure of the bridge network and the road; Wherein, the comprehensive reward function based on maintenance cost, asset risk and traffic network capacity loss risk is represented by the following formula (1): Reward = (C M + R AST ) / λ ECON + w NET R NET / λ NET (1) wherein R AST represents the asset risk corresponding to the bridge state, which is represented by the following equation (2): R AST =∑p f (s)C f (2) wherein R NET represents the risk of loss of traffic capacity of the bridge network corresponding to the bridge state, and is represented by the following equation (3): R NET =∑p(c|)[F0-F(c)] (3) wherein, Reward represents the comprehensive reward function; C M represents the direct maintenance cost; λ ECON represents the direct maintenance cost and the asset risk scaling coefficient; w NET represents the bridge network traffic capacity loss risk weight coefficient; λ NET represents the direct maintenance cost and the asset risk scaling coefficient; p f (s) represents the failure probability corresponding to the bridge state; C f represents the asset failure cost; p(c|s) represents the bridge failure combination corresponding to the bridge state; [F0-F(c)] represents the bridge network traffic capacity loss under the bridge failure combination; S3, constructing a bridge maintenance decision problem according to the state space of the bridge, the maintenance action space, the state transition matrix and the comprehensive reward function; S4, describing the bridge maintenance decision problem as a Markov decision process, establishing a pointer network policy model using a pointer network, training the pointer network policy model using an Actor-Critic algorithm, and obtaining a maintenance decision model based on reinforcement learning; training the maintenance decision model based on reinforcement learning until convergence, and outputting a bridge maintenance action sequence under limited constraints, including: S41, using a pointer network to construct a pointer policy network model, inputting bridge attributes, bridge states and maintenance actions into an encoder to generate feature codes; using a decoder to integrate feature codes, current bridge information and traversed partial bridge numbers, and generating the bridge and maintenance action with the largest weight as the next step maintenance bridge and bridge maintenance action through an attention mechanism; calculating the preliminary maintenance cost according to the bridge and bridge maintenance action; S42, constructing an Actor-Critic network according to the pointer policy network model; wherein the Actor-Critic network comprises a pointer policy network and a value network; wherein the pointer policy network is used to output bridge maintenance priority and maintenance action policy according to the bridge state; wherein the value network is used to predict state value according to the bridge state and the information processed by the encoder; S43, constructing an advantage function; constructing a pointer policy network loss function and a value network loss function according to the advantage function; S44, repairing according to the bridge maintenance priority sequence generated by the pointer policy network to obtain an actual reward value; taking the mean square error of the actual reward value and the estimated reward value generated by the value network as an optimization objective, training the Actor-Critic network using the gradient descent method through the pointer policy network loss function and the value network loss function until convergence, and outputting a bridge maintenance action sequence under limited constraints.

2. The method of claim 1, wherein, The state space of the bridge is represented by the following formula (4): Wherein, S represents the state space of the bridge; Wherein, the maintenance action space is represented by the following formula (5): Wherein, A represents the maintenance action space; Wherein, the state transition matrix is represented by the following formula (6): Wherein, P represents the state transition matrix.

3. The method of claim 1, wherein, The advantage function is represented by the following formula (7): Adv(s,a)=r+γV(s′)-V(s) (7) Wherein, r represents the reward obtained at the current time step; γ represents the discount factor; V(s') represents the estimated value of the next state by the value network; V(s) represents the estimated value of the current state by the value network; Adv(s, a) is the advantage function, representing the return of performing action a in state s; s represents the current state of the bridge; a represents the maintenance action taken.

4. The method of claim 1, wherein, The loss function of the pointer policy network is represented by the following formula (8): where ActorLoss represents the loss of the actor policy network; E represents an expectation operation; denotes the log gradient of the policy function π(a|s) with respect to its implicit parameters θ; where π(a|s) represents the probability of taking action a in state s.

5. The method of claim 1, wherein, The loss function of the value network is represented by the following formula (9): CriticLoss = E[Adv(s, a) 2 ] (9) Wherein, CriticLoss represents the loss of the value network.

6. A bridge group maintenance priority dynamic decision device based on reinforcement learning, the bridge group maintenance priority dynamic decision device is used to implement the bridge group maintenance priority dynamic decision method based on reinforcement learning as claimed in any one of claims 1-5, characterized in that, The device comprises: A first construction unit for constructing the topology of the bridge network and the road; defining the state space, the maintenance action space and the state transition matrix of the bridge; A design unit for defining the reliability index corresponding to the state of the bridge, and designing a comprehensive reward function based on maintenance cost, asset risk and traffic network capacity loss risk based on the topology of the bridge network and the road; A second construction unit for constructing the bridge maintenance decision problem according to the state space, the maintenance action space, the state transition matrix of the bridge and the comprehensive reward function; A training unit for describing the bridge maintenance decision problem as a Markov decision process, establishing a pointer network policy model using a pointer network, training the pointer network policy model using an Actor-Critic algorithm, and establishing a maintenance decision model based on reinforcement learning; training the maintenance decision model based on reinforcement learning until convergence, and outputting the bridge maintenance action ranking under limited constraints.

7. A bridge group maintenance priority dynamic decision device based on reinforcement learning, characterized in that, The bridge group maintenance priority dynamic decision device based on reinforcement learning comprises: A processor; A memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method of any one of claims 1 to 5.

8. A computer readable storage medium, characterized in that, The computer readable storage medium stores program code that can be called and executed by the processor to implement the method of any one of claims 1 to 5.

Images