GA-RL-based hierarchical multi-agent bridge-tunnel group maintenance decision method and device

Abstract

The application provides a layered multi-agent bridge and tunnel group maintenance decision method and device based on GA-RL, and relates to the technical field of civil engineering.The method comprises the following steps: constructing an expert strategy library according to an expert strategy and a genetic algorithm offline, and selecting an optimal strategy online according to a network-level reinforcement learning agent;training a bridge and tunnel-level reinforcement learning agent dedicated to each type of bridge and tunnel by using a proximal policy optimization algorithm, and generating a maintenance application of a bridge and tunnel to be maintained; converting the optimal strategy into a computable priority evaluation model to generate a bridge and tunnel maintenance priority sequence, optimizing a budget allocation ratio according to a differential evolution algorithm, and generating a network-level maintenance scheme for the bridge and tunnel group.The layered multi-agent decision framework based on the GA-RL hybrid algorithm proposed by the application not only makes up for the lack of environmental adaptability of static optimization, but also breaks through the calculation bottleneck of dynamic programming in large-scale scenarios, and provides a more efficient and adaptive solution for complex bridge and tunnel group network maintenance decision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of civil engineering and artificial intelligence, and particularly relates to a layered multi-agent bridge tunnel group maintenance decision-making method and device based on GA-RL. BACKGROUND

[0002] As the core node of the traffic network, the maintenance decision of the bridge directly affects the operation efficiency and public safety of the road network, and needs to take into account the structural safety, resource efficiency and network resilience. With the acceleration of infrastructure aging and the intensification of disaster risks, the traditional single bridge maintenance mode has been difficult to meet the complex needs: on the one hand, single decision-making easily leads to unbalanced resource allocation, such as over-maintenance of key bridges and insufficient maintenance of non-critical bridges; on the other hand, the passive repair mode lags behind the deterioration process and may cause a chain of safety risks. In recent years, the maintenance strategy is undergoing a transformation from "single isolation" to "group coordination" and from "passive response" to "active prevention". Typical researches such as bridge network multi-objective optimization and life cycle cost analysis all emphasize improving the scientific nature of decision-making through systematic thinking. However, the existing technologies mainly focus on static scenarios, which are difficult to dynamically adapt to the nonlinear characteristics of bridge deterioration and the randomness of environmental interference, resulting in problems such as insufficient adaptability and low computational efficiency of the maintenance strategy in practical applications.

[0003] The current mainstream technical solutions can be divided into two categories: one is static planning based on optimization algorithms, such as using GA (Genetic Algorithm) to optimize the multi-objective of maintenance cost and reliability index, or using machine learning to build a deterioration prediction model to assist decision-making; the other is single decision-making based on dynamic programming, such as using MDP (Markov Decision Process) or RL (Reinforcement Learning) to realize dynamic adjustment of the maintenance time. However, the former usually assumes that the deterioration process is in a steady state, and does not consider the resource competition and coordination constraints between bridge groups, which makes the optimization results difficult to implement in large-scale networks; the latter has dynamic adaptability, but is mainly designed for single structure, and is prone to state space explosion when dealing with complex scenarios of multiple bridges and multiple constraints, making the calculation infeasible.

[0004] In summary, the existing technologies either ignore the dynamic deterioration characteristics in static optimization or are limited in dynamic programming due to the ineffective solution of resource coordination and computational efficiency in large-scale networks.

[0005] In the field of bridge and tunnel infrastructure maintenance, existing technologies have formed a dynamic programming framework based on DRL (Deep Reinforcement Learning), whose core architecture can be summarized as an "Actor-Critic" network model architecture. The method generates maintenance actions (such as no action, maintenance, repair, reconstruction, or replacement) through the Actor network, and the Critic network evaluates the value of the actions. The state space is defined as the state vectors of key elements of all bridges, and the action space is defined as no repair, routine maintenance, repair, restoration, and replacement. The dimension of the action space grows exponentially with the number of bridges.

[0006] Although deep reinforcement learning offers a new solution for dynamic maintenance decision-making, it still suffers from two fundamental drawbacks in practical applications, directly limiting its effectiveness in large-scale urban bridge and tunnel networks:

[0007] (1) The explosion of dimensions in the state space and action space makes computation infeasible:

[0008] Existing deep reinforcement learning methods mostly employ a single-agent architecture, whose action space grows exponentially with the number of bridges. For example, for... The maintenance decisions for this bridge involve a high degree of complexity in the operational space. ( (This refers to the number of possible actions for a single bridge). Although some studies have attempted to extend this to multi-agent algorithms to achieve distributed decision-making, these methods still require a central node to integrate global state information, leading to a non-linear increase in communication overhead and computational complexity with network size. In networks with hundreds of bridges, traditional DRL frameworks struggle to achieve real-time decision-making due to the explosion of the state-action space, severely limiting their scalability.

[0009] (2) The failure of the learning mechanism under rigid constraints leads to policy bias:

[0010] Actual bridge maintenance projects involve multiple types and quantities of rigid constraints (such as budget limits and risk thresholds). Traditional RL trial-and-error learning mechanisms are prone to dilemmas in this scenario: On the one hand, the penalty function method guides policy learning by introducing a penalty term for constraint violation, but the static setting of the penalty coefficient lacks theoretical guidance, which can easily lead to overly conservative policies (such as sacrificing maintenance effectiveness to meet the budget) or constraint violations (such as cost overruns); on the other hand, although the action masking mechanism can shield some illegal actions, in long-term sequential decision-making, the constraint boundary has dynamic ambiguity, making it difficult to clearly define the "illegal model", and the masking operation will compress the policy exploration space, hindering the discovery of the globally optimal policy.

[0011] In summary, the existing DRL maintenance decision-making method is only applicable to small-scale and simplified scenarios without rigid constraints, but in a large-scale bridge and tunnel network at the city level, due to the dual challenges of state space explosion and constraint learning failure, the decision-making performance is sharply reduced and cannot meet the actual engineering requirements. SUMMARY

[0012] In order to solve the technical problems of state space and action space dimension explosion leading to infeasible calculation and learning mechanism failure under rigid constraints causing strategy deviation existing in the prior art, an embodiment of the present application provides a layered multi-agent bridge and tunnel group maintenance decision-making method and device based on GA-RL. The technical solution is as follows:

[0013] On the one hand, a layered multi-agent bridge and tunnel group maintenance decision-making method based on GA-RL is provided, which is realized by a layered multi-agent large-scale bridge and tunnel group maintenance decision-making device, and the method comprises:

[0014] S1, obtaining data information and expert strategy of a bridge and tunnel group in a road network.

[0015] S2, constructing an expert strategy library offline according to the expert strategy and a genetic algorithm, and selecting an optimal strategy online according to the expert strategy library and a network-level reinforcement learning agent.

[0016] S3, training a dedicated bridge and tunnel-level reinforcement learning agent for each type of bridge and tunnel by using a proximal policy optimization algorithm, constructing a bridge and tunnel agent cluster, and generating a maintenance application for the bridge and tunnel to be maintained by the bridge and tunnel-level reinforcement learning agent.

[0017] S4, converting the optimal strategy into a computable priority evaluation model according to the optimal strategy, the maintenance application and the obtained data information, generating a bridge and tunnel maintenance priority sequence according to the priority evaluation model, optimizing a budget allocation ratio according to the bridge and tunnel maintenance priority sequence and a differential evolution algorithm, generating a bridge and tunnel group network-level maintenance scheme, and executing the maintenance scheme by the bridge and tunnel-level reinforcement learning agent.

[0018] Optionally, the construction of the expert strategy library offline according to the expert strategy and the genetic algorithm and the selection of the optimal strategy online according to the expert strategy library and the network-level reinforcement learning agent in S2 comprise:

[0019] S21, formalizing the expert strategy into a strategy vector, as shown in the following formula (1):

[0020] (1)

[0021] In the formula, denotes the strategy vector, denotes the weight vector, denotes the number of preferences in the expert strategy, denotes the threshold vector, The number of threshold constraints in the expert strategy.

[0022] S22, mapping the strategy vector to the gene bit by using the fixed-length vector of real value coding to obtain a chromosome, as shown in the following formula (2):

[0023] (2)

[0024] In the formula, The chromosome is represented.

[0025] S23, generating an initial population according to the chromosome and Latin hypercube sampling.

[0026] S24, designing an evolutionary operator to generate new strategies in the solution space according to the initial population and the evolutionary operator.

[0027] S25, selecting the top-K optimal strategies, and offline constructing an expert strategy library; wherein K is a preset value.

[0028] S26, constructing an action space according to the expert strategy and the top-K new strategies, and selecting an optimal strategy from the action space by a network-level reinforcement learning agent.

[0029] Optionally, the selection operator of the evolutionary operator in S24 uses network health as the fitness of the genetic algorithm, and combines roulette selection and elite reservation strategy.

[0030] The crossover operator introduces boundary protection to simulate binary crossover.

[0031] The mutation operator uses Gaussian disturbance and non-uniform mutation.

[0032] Optionally, the state space of the network-level reinforcement learning agent in S26 is: the network health of the bridge and tunnel agent cluster, the annual maintenance budget, the bridge and tunnel to be maintained, and the optimal maintenance action suggestion cost and the alternative maintenance action suggestion cost of the bridge and tunnel to be maintained.

[0033] The reward function of the network-level reinforcement learning agent is: maximizing the network health.

[0034] Optionally, the proximal policy optimization algorithm in S3 trains a dedicated bridge and tunnel-level reinforcement learning agent for each type of bridge and tunnel to construct a bridge and tunnel agent cluster, and generates a maintenance application for the bridge and tunnel to be maintained by the bridge and tunnel-level reinforcement learning agent, including:

[0035] S31, a proximal policy optimization algorithm is used to train a bridge-level reinforcement learning agent specific to each bridge tunnel type, and bridge-level reinforcement learning agents of multiple bridge tunnel types are integrated to obtain a bridge tunnel agent cluster; wherein all different bridge tunnel instances serving the same bridge tunnel type share the same trained bridge-level reinforcement learning agent and parameters.

[0036] S32, the bridge-level reinforcement learning agent calculates the probability distribution of all feasible maintenance actions under the current state of the bridge tunnel to be maintained, and the maintenance action with the highest probability value is recommended as the optimal maintenance action, and the maintenance action with the second highest probability value is recommended as the alternative maintenance action.

[0037] S33, generating a maintenance application according to the optimal maintenance action suggestion and the alternative maintenance action suggestion.

[0038] Optionally, the parameter update in S31 follows the collective optimization principle, as shown in the following formula (3):

[0039] (3)

[0040] In the formula, denotes the policy network parameters corresponding to the bridge tunnel type, denotes the learning rate parameter, denotes the number of bridge tunnel types, is the sum of the gradients of all bridge tunnel instances of the same bridge tunnel type, and denotes the collective optimization direction, denotes the parameters of the policy network.

[0041] Optionally, the mutation operation in the differential evolution algorithm in S4 is as shown in the following formula (4):

[0042] (4)

[0043] In the formula, denotes the mutated individual, denotes the index of the target individual in the population, , , denotes three different individuals selected at random, denotes the scaling factor.

[0044] On the other hand, a GA-RL-based hierarchical multi-agent bridge tunnel group maintenance decision device is provided, which is applied to a GA-RL-based hierarchical multi-agent bridge tunnel group maintenance decision method, and the device comprises:

[0045] A data acquisition module for acquiring data information and expert strategies of a road network bridge tunnel group.

[0046] a network layer module, configured to construct an expert strategy library offline according to the expert strategy and a genetic algorithm, and select an optimal strategy online according to the expert strategy library and a network-level reinforcement learning agent.

[0047] a bridge layer module, configured to train a bridge-level reinforcement learning agent dedicated to each bridge type by using a proximal policy optimization algorithm, construct a bridge agent cluster, and generate a maintenance application for a bridge to be maintained by the bridge-level reinforcement learning agent.

[0048] an intermediate layer module, configured to convert the optimal strategy into a calculable priority evaluation model according to the optimal strategy, the maintenance application and obtained data information, generate a bridge maintenance priority sequence according to the priority evaluation model, optimize a budget allocation ratio according to the bridge maintenance priority sequence and a differential evolution algorithm, generate a bridge group network-level maintenance scheme, and execute the maintenance scheme by the bridge-level reinforcement learning agent.

[0049] Optionally, the network layer module is further configured to:

[0050] S21. formalize the expert strategy into a strategy vector, as shown in the following formula (1):

[0051] (1)

[0052] wherein, denotes the strategy vector, denotes a weight vector, denotes a number of preferences in the expert strategy, denotes a threshold vector, denotes a number of threshold constraints in the expert strategy.

[0053] S22. map the strategy vector to a gene bit by using a fixed-length vector with real value coding to obtain a chromosome, as shown in the following formula (2):

[0054] (2)

[0055] wherein, denotes the chromosome.

[0056] S23. generate an initial population according to the chromosome and Latin hypercube sampling.

[0057] S24. design an evolution operator to generate a new strategy in a solution space according to the initial population and the evolution operator.

[0058] S25. select a TOP-K number of new strategies with optimal performance to construct the expert strategy library offline; wherein K is a preset value.

[0059] S26, constructing an action space according to the expert strategy and the TOP-K new strategies, and selecting an optimal strategy from the action space by the network-level reinforcement learning agent online.

[0060] Optionally, the selection operator of the evolution operator adopts the network health degree as the fitness of the genetic algorithm, and combines the roulette selection and the elite reservation strategy.

[0061] The crossover operator introduces boundary protection and simulates binary crossover.

[0062] The mutation operator adopts Gaussian disturbance and non-uniform mutation.

[0063] Optionally, the state space of the network-level reinforcement learning agent is: the network health degree of the bridge and tunnel agent cluster, the annual maintenance budget, the bridge and tunnel to be maintained, and the optimal maintenance action recommendation cost and the alternative maintenance action recommendation cost of the bridge and tunnel to be maintained.

[0064] The reward function of the network-level reinforcement learning agent is: maximizing the network health degree.

[0065] Optionally, the bridge and tunnel layer module is further used for:

[0066] S31, using a proximal policy optimization algorithm to train a dedicated bridge and tunnel-level reinforcement learning agent for each bridge and tunnel type, integrating bridge and tunnel-level reinforcement learning agents of multiple bridge and tunnel types to obtain a bridge and tunnel agent cluster; wherein different bridge and tunnel instances serving the same bridge and tunnel type share the same trained bridge and tunnel-level reinforcement learning agent and parameters.

[0067] S32, calculating the probability distribution of all feasible maintenance actions under the current state of the bridge and tunnel to be maintained by the bridge and tunnel-level reinforcement learning agent, taking the maintenance action with the highest probability value as the optimal maintenance action recommendation, and taking the maintenance action with the second highest probability value as the alternative maintenance action recommendation.

[0068] S33, generating a maintenance application according to the optimal maintenance action recommendation and the alternative maintenance action recommendation.

[0069] Optionally, the parameter update follows the collective optimization principle, as shown in the following formula (3):

[0070] (3)

[0071] In the formula, denotes the strategy network parameter corresponding to the bridge and tunnel type, denotes the learning rate parameter, denotes the number of bridge and tunnel types, is the sum of the gradients of all bridge and tunnel instances of the same bridge and tunnel type, and denotes the collective optimization direction, denotes the parameter of the strategy network.

[0072] Optionally, the mutation operation in the differential evolution algorithm is as shown in the following formula (4):

[0073] (4)

[0074] In the formula, denotes a mutated individual, denotes the index of the target individual in the population, , , denotes three different individuals selected at random, denotes a scaling factor.

[0075] In another aspect, a hierarchical multi-agent large-scale bridge and tunnel group maintenance decision device is provided, and the hierarchical multi-agent large-scale bridge and tunnel group maintenance decision device comprises a processor and a memory, wherein the memory has computer readable instructions stored thereon, and the computer readable instructions are executed by the processor to implement any one of the above GA-RL based hierarchical multi-agent bridge and tunnel group maintenance decision methods.

[0076] In another aspect, a computer readable storage medium is provided, and the storage medium has at least one instruction stored therein, and the at least one instruction is loaded and executed by a processor to implement any one of the above GA-RL based hierarchical multi-agent bridge and tunnel group maintenance decision methods.

[0077] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0078] In the present application, the calculation bottleneck of dimension explosion of state space and action space is broken through: traditional single-agent RL is difficult to make decisions in a hundred-seat bridge and tunnel network due to the exponential growth of action space (N , is the number of single bridge actions, is the number of bridges). The present application reduces the calculation complexity to by decoupling global strategy generation and local decision optimization (bridge and tunnel) through a hierarchical architecture. Network-level agents compress the global action space from a space to the number of strategies in the strategy library .

[0079] Adaptive policy learning under rigid constraints: traditional penalty function method is prone to conservative or default policies due to the difficulty in setting the penalty coefficient. The present application converts the total budget into a resource allocation optimization problem through the coordination layer, and uses the differential evolution algorithm for resource allocation to ensure that the final decision scheme strictly complies with the constraints.

[0080] Flexible expansion capability of bridge and tunnel type and scale: the application constructs a full-dimensional scalable decision framework through the mechanism of dedicated type training of the bottom intelligent agent cluster and parameter sharing deployment, and realizes the flexible expansion capability of the bridge and tunnel type and quantity. In the type dimension, independent PPO intelligent agents are trained for different bridge and tunnel types such as beam bridges, arch bridges and cable-stayed bridges, to ensure that the decision accurately adapts to the deterioration mechanism of various bridges and tunnels; when a new type (such as a rigid frame bridge) is added, only the corresponding intelligent agent needs to be trained to seamlessly expand. In the quantity dimension, through the parameter sharing mechanism, bridges and tunnels of the same type share the same model parameters, avoiding the repeated creation of independent models for each bridge and tunnel, and reducing the consumption of computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0082] Figure 1 is a hierarchical multi-agent large-scale bridge and tunnel group maintenance decision method flowchart provided by the embodiment of the present application;

[0083] Figure 2 is a hierarchical multi-agent bridge and tunnel group maintenance decision system architecture diagram provided by the embodiment of the present application;

[0084] Figure 3 is a GA-RL hierarchical multi-agent bridge and tunnel maintenance decision flowchart provided by the embodiment of the present application;

[0085] Figure 4 is a genetic algorithm flowchart based on expert knowledge guidance provided by the embodiment of the present application;

[0086] Figure 5 is a hierarchical multi-agent large-scale bridge and tunnel group maintenance decision device block diagram based on GA-RL provided by the embodiment of the present application;

[0087] Figure 6 is a structural schematic diagram of a large-scale hierarchical multi-agent bridge and tunnel group maintenance decision device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0088] The technical solutions in the present application will be described below with reference to the drawings.

[0089] In the embodiments of the present application, the words such as "example", "for example" are used to represent an example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0090] In the embodiments of the present application, "image" and "picture" can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized. "Of", "corresponding" and "corresponding" can be used interchangeably at times, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized.

[0091] In the embodiments of the present application, sometimes the subscript such as W1 can be written in the form of non-subscript such as W1, and the meanings expressed are consistent when the distinction is not emphasized.

[0092] In order to make the technical problems, technical schemes and advantages to be solved by the present application more clear, the following will be described in detail in combination with the drawings and specific embodiments.

[0093] The embodiment of the present application provides a layered multi-agent bridge and tunnel group maintenance decision method based on GA-RL, which can be realized by a layered multi-agent large-scale bridge and tunnel group maintenance decision device. The layered multi-agent large-scale bridge and tunnel group maintenance decision device can be a terminal or a server. Figure 1 As shown in the flow chart of the layered multi-agent large-scale bridge and tunnel group maintenance decision method based on GA-RL, the processing flow of the method can include the following steps:

[0094] S1, obtaining data information and expert strategy of a road network bridge and tunnel group.

[0095] In a feasible implementation manner, the data information of the bridges and tunnels of the road network bridge and tunnel group is obtained, wherein the bridge and tunnel type information is taken as the classification basis of the bridge and tunnel agent cluster; and the construction cost, state change history data (state transition matrix) and other information of the bridge and tunnel are used to construct the environment for training the exclusive PPO agent. The second is expert knowledge input, wherein in the offline stage of the network layer, the GA algorithm generates a high-quality strategy library through expert knowledge.

[0096] The layered multi-agent bridge and tunnel maintenance decision based on GA-RL adopts a three-level collaborative architecture of "network layer-intermediate layer-bridge and tunnel layer" as shown in Figure 2 Through function decoupling and multi-agent cooperation, closed-loop control from global strategy generation to local decision optimization is realized.

[0097] S2, constructing an expert strategy library according to the expert strategy and the genetic algorithm offline, and selecting an optimal strategy online according to the expert strategy library and the network-level reinforcement learning agent.

[0098] In an implementable embodiment, the expert knowledge guided genetic algorithm is the core engine of global strategy generation in the framework, and through the structured embedding of domain expert experience and adaptive evolution mechanism, efficient construction of a high-quality strategy library is realized. The core implementation includes the following three stages: mathematical representation of expert strategy, chromosome coding and initial population generation, and evolution operator design, and the specific steps include:

[0099] S21, first formalize the experience knowledge of the domain expert into a strategy vector, as shown in the following formula (1):

[0100] (1)

[0101] In the formula, denotes the strategy vector, and the weight vector denotes the relative importance of different decision dimensions (such as safety, cost, risk, etc.), denotes the number of preferences in the expert strategy, for example, an expert strategy , considering 3 factors, which are cost, network health, and bridge state level, then , corresponding to is the cost weight, is the network health weight, is the state level weight, and the threshold vector defines the critical condition of state transition or decision trigger (such as degradation threshold, risk boundary), denotes the number of threshold constraints in the expert strategy, for example, an expert strategy considers two threshold constraints, then , the bridge corresponding to state 4 is not allowed on the bridge, , and the network capacity of the bridge-tunnel network is not allowed to be lower than 3000 .

[0102] S22, in the chromosome coding and initial population generation stage, the algorithm uses a fixed-length vector with real value coding to realize the mapping of strategy parameters to gene bits; each chromosome is composed of continuously arranged weight gene segments and threshold gene segments, and the gene bit value directly corresponds to the specific parameters of the strategy vector, as shown in the following formula (2):

[0103] (2)

[0104] In the formula, denotes the chromosome, the first bit encodes the weight vector , the back bit encoding threshold vector , forming a gene chain with a length of .

[0105] S23, the generation of the initial population combines expert experience and random exploration: expert experience individuals are generated by manually setting initial parameters by domain experts, ensuring that the initial population contains effective solutions of expert experience; the remaining individuals are randomly generated in the parameter space using Latin hypercube sampling, maintaining population diversity. This hybrid initialization strategy avoids the low efficiency search caused by pure random method, and prevents the strategy homogeneity caused by expert experience limitations, laying a good genetic foundation for the subsequent evolution process.

[0106] S24, the design of evolution operators embodies the deep adaptation of the algorithm to the special needs of bridge and tunnel maintenance. The selection operator adopts an elite reservation strategy based on network health degree fitness. The network health degree of each chromosome corresponding to the strategy is calculated as the fitness of the genetic algorithm. Through the combination of roulette selection and elite reservation mechanism, the survival probability of high fitness individuals is guaranteed, and the evolution pressure of the population is maintained.

[0107] The crossover operator introduces boundary protection SBX (Simulated Binary Crossover), which controls the parameter distribution range of the offspring by a probability density function when mixing parameters between parent chromosomes, preventing the generation of abnormal strategies that violate engineering common sense.

[0108] The mutation operator re-normalizes the weight gene segment after Gaussian perturbation, ensuring that the weight vector always satisfies the constraint condition . The threshold gene segment implements non-uniform mutation, and the mutation amplitude decays with the increase of iteration number, balancing global exploration and local fine search.

[0109] S25, select the top-K new strategies with the best performance to offline build an expert strategy library; where K is a preset value.

[0110] S26, construct an action space according to the expert strategy and the top-K new strategies, and select the optimal strategy from the action space by a network-level reinforcement learning agent online; where the state space of the network-level reinforcement learning agent is: the network health degree of the bridge and tunnel agent cluster, the annual maintenance budget, the bridge and tunnel to be maintained, and the optimal maintenance action suggestion cost and the alternative maintenance action suggestion cost of the bridge and tunnel to be maintained; the reward function of the network-level reinforcement learning agent is: maximizing the network health degree.

[0111] In a feasible implementation, at the system architecture design level, the top layer is a network-level agent, which adopts a two-stage separation design mechanism as a global decision unit to clearly divide the decision-making process into two independent stages of offline strategy generation and online strategy selection, to ensure the efficiency and adaptability of decision-making.

[0112] In the offline stage, the construction of the strategy library mainly relies on genetic algorithm. This process is guided by domain expert knowledge, which is abstracted as a set of weights with specific semantic orientation. Specifically, expert strategies can be formally represented as a combination of weighted parameters, with each type of strategy representing different decision preferences (such as state priority, cost optimization, etc.). GA uses these expert strategies as initial guidance or reference solutions to perform global exploration in the solution space through selection, crossover, mutation, and other evolutionary operations to generate a set of diverse high-quality strategy solutions. After optimization, the system retains the TOP-K strategies with the best performance during the GA search process (K is a preset value) to form a pre-generated strategy pool.

[0113] Further, in the online stage, the dynamic strategy selection is entrusted to the RL agent. The action space of the upper RL is defined as discrete fixed strategy options. These options are directly derived from the results of the offline stage: they include both the original strategy weights provided by experts and the TOP-K optimized strategy solutions generated by GA. For example, if an expert strategy is introduced at system initialization and K = 5, the RL action space will include these 6 fixed strategy combinations (1 expert strategy + 5 GA strategies). The goal of RL is to learn to select the strategy solution that best optimizes the current global objective from this set of predefined strategies through continuous interaction with the environment.

[0114] This design achieves dual advantages through hierarchical decoupling: GA in the offline stage fully utilizes expert prior knowledge and leverages its global search capability to efficiently generate a strategy pool; RL in the online stage focuses on efficient and robust strategy switching and deployment from limited but high-quality strategy options based on real-time feedback in a dynamically changing operating environment. The two stages work closely together to ultimately endow the network-level agent with intelligent decision-making capabilities that balance global resource allocation, risk prevention and control, and long-term performance optimization.

[0115] S3, adopt a proximal policy optimization algorithm to train a bridge-tunnel level reinforcement learning agent for each type of bridge-tunnel, construct a bridge-tunnel agent cluster, and generate a maintenance application for the bridge-tunnel to be maintained by the bridge-tunnel level reinforcement learning agent.

[0116] Optionally, the above step S3 can include the following steps S31-S33:

[0117] S31, a proximal policy optimization algorithm is used to train a bridge-level reinforcement learning agent specific to each bridge type, and bridge-level reinforcement learning agents of multiple bridge types are integrated to obtain a bridge agent cluster; wherein all different bridge instances serving the same bridge type share the same trained bridge-level reinforcement learning agent and parameters.

[0118] In a feasible implementation, the bottom layer is a bridge-level RL agent cluster. Considering that there are a large number of bridge facilities with different structures (such as beam bridges, arch bridges, cable-stayed bridges, etc.) in actual bridge networks, and the deterioration mechanism and maintenance requirements of these bridge facilities differ significantly, traditional single decision models are difficult to effectively cope with. In order to ensure that each specific type of bridge can obtain accurate and professional maintenance decisions, while maintaining the scalability of the entire framework to the size of the bridge, the present application trains a dedicated reinforcement learning agent for each determined bridge type.

[0119] Specifically, PPO (Proximal Policy Optimization) is a deep reinforcement learning algorithm based on policy gradient. Its core design philosophy is to balance the exploration efficiency and algorithm stability by limiting the update amplitude of the policy. Compared with the traditional policy gradient method, which is prone to policy collapse due to excessive update steps, PPO introduces a clipping mechanism to soft-constrain the policy update. Its objective function can be expressed as formula (3):

[0120] (3)

[0121] In the formula, represents the objective function, represents the policy network parameters, represents the expectation of time step , represents the current policy, represents the old policy, represents the advantage function estimate value, represents the clipping threshold. This formula limits the probability ratio to be within the interval , effectively preventing the problem of excessive single-step updates.

[0122] ​In the training phase, for each type of bridge and tunnel, a dedicated agent is independently trained using the PPO algorithm. Each dedicated agent can make accurate and professional maintenance decisions for the specific bridge and tunnel type it is responsible for. Secondly, in the deployment phase, these trained and dedicated PPO agents are integrated into a unified cluster through a network parameter sharing mechanism. When the framework is actually running, each specific bridge and tunnel in the network is dynamically assigned to the corresponding trained agent instance based on its bridge and tunnel type. The key point is that parameter sharing here mainly reflects the flexibility of cluster deployment: all different bridge and tunnel instances serving the same type of bridge and tunnel share the same trained agent model and all its parameters. This approach avoids the huge overhead of repeatedly creating independent models for a large number of the same type of bridge and tunnel in the network, greatly improving resource utilization efficiency and system efficiency. At the same time, it significantly simplifies the dynamic expansion of the cluster. When a new bridge and tunnel type is introduced into the network, only a new agent for that type needs to be trained and deployed to the cluster to serve all instances of that type.

[0123] Further, the parameter sharing technique is an architecture design method that achieves efficient resource utilization by reusing neural network weights. Its core idea is to abstract the decision logic of similar tasks into a shared parameter space, reducing system redundancy. This technique achieves weight reuse through a horizontal parameter sharing mechanism: for instances of the same type of task, the same strategy network and value network parameters are shared, reducing the model storage requirement from a linear growth pattern (linear expansion with the number of instances) to a constant level of overhead (only one set of core parameters needs to be maintained). This design avoids the memory explosion caused by full model replication, while ensuring professional decision-making capabilities and providing a scalable solution for large-scale task clusters.

[0124] In the bridge and tunnel agent cluster of the present invention, the parameter sharing technique achieves efficient deployment through a hierarchical architecture. For the same type of bridge and tunnel (such as a beam bridge), all instances share a pre-trained strategy network and value network , whose parameter updates follow the principle of collective optimization:

[0125] (4)

[0126] Equation (4) represents the optimization process of the strategy parameters of the underlying multi-type bridge and tunnel agent cluster. The goal of the equation is to maximize the expected return through gradient ascent. Where represents the strategy network parameters of the type of bridge and tunnel, represents the learning rate parameter, represents the number of bridge and tunnel types, To sum up the gradients of all bridge tunnel instances of the same type, the optimization direction of the cluster is represented, and cross-type knowledge sharing is achieved by aggregating and averaging the gradients of all bridge tunnel types. Parameters representing the policy network.

[0127] When a new bridge tunnel type is added, the system independently trains dedicated parameters and registers them to the cluster, and existing type instances can seamlessly access the shared parameter pool. At runtime, each agent instance realizes state individualization by injecting a unique identifier, and the shared network outputs an action probability distribution based on the current state of the instance.

[0128] S32, calculate the probability distribution of all feasible maintenance actions of the bridge tunnel under the current state of the bridge tunnel to be maintained by the bridge tunnel level reinforcement learning agent, and take the maintenance action with the highest probability value as the optimal maintenance action suggestion and the maintenance action with the second highest probability value as the alternative maintenance action suggestion.

[0129] In an available implementation, in the execution phase, the bridge tunnel level RL agent makes maintenance decisions based on its own state, and when the assigned bridge tunnel needs maintenance, the agent determines the optimal maintenance action suggestion as the action with the highest probability value and the alternative maintenance action suggestion as the action with the second highest probability value based on the probability distribution of all feasible maintenance actions under the current state.

[0130] S33, generate a maintenance application according to the optimal maintenance action suggestion and the alternative maintenance action suggestion.

[0131] In an available implementation, a maintenance application containing information such as bridge tunnel identification, suggested action and its cost is generated and submitted to the upper coordination layer. After the bridge tunnel level agent that submitted the application receives the final maintenance scheme issued by the coordination layer, it will no longer execute the optimal or alternative action suggestion it initially proposed, but will strictly execute the final scheme optimized by the coordination layer to complete the task closed loop as the end execution unit.

[0132] In the present application, PPO algorithm is used to train dedicated reinforcement learning agents for each type of bridge tunnel. The agent takes the state data of the bridge tunnel as input, and the reward function takes the structural risk and maintenance cost as optimization objectives. The action probability distribution of the maintenance is output through the policy network, the action with the highest probability is selected as the optimal action, and the action with the second highest probability is selected as the alternative action. The primary and alternative maintenance action suggestions are finally output for the upper coordination layer to perform global optimization configuration. In the network layer, PPO algorithm is used to train the policy selection agent, which learns to select the strategy scheme that can best optimize the current global objective from the set of predefined strategies through continuous interaction with the environment.

[0133] S4, according to the optimal strategy, the maintenance application and the obtained data information, the optimal strategy is converted into a computable priority evaluation model, a bridge tunnel maintenance priority sequence is generated according to the priority evaluation model, a budget allocation ratio is optimized according to the bridge tunnel maintenance priority sequence and the differential evolution algorithm, a maintenance scheme is generated, and the maintenance scheme is executed by the bridge tunnel level reinforcement learning agent.

[0134] In an available implementation, the intermediate layer-coordination layer (referred to as the coordination layer) is a key hub connecting the upper layer strategy decision and the bottom layer execution, and the core function thereof is to realize the optimal allocation of global resources through dynamic scheduling and conflict resolution. It first aggregates three types of key inputs: the maintenance application submitted by the bridge tunnel layer agent (including the bridge tunnel state, the optimal and alternative maintenance scheme and the cost), the strategy weight vector issued by the upper layer network level agent, and the system level hard constraint (such as the total budget). By analyzing the strategy weight, the coordination layer converts the weight strategy into a computable priority evaluation model. The priority evaluation model generates a bridge tunnel maintenance priority sequence, and based on the priority sequence, the coordination layer allocates resources by using the differential evolution algorithm.

[0135] Under the guidance of the upper layer strategy, the coordination layer realizes the optimal allocation of resources and the flexible degradation of maintenance actions through maintenance priority evaluation and budget allocation, which not only guarantees the accurate implementation of the upper layer strategy, but also gives the bottom layer operation sufficient flexibility, thereby constituting the core hub of the framework collaborative optimization.

[0136] Specifically, the differential evolution algorithm is a global optimization algorithm based on swarm intelligence, which realizes the search of the solution space by simulating the mutation, crossover and competition mechanism in natural selection. The algorithm first randomly initializes a group of solution vectors to form an initial population, and each solution represents a potential optimal configuration of the problem. In the iteration process, mutation operation is performed on each solution vector: three different individuals are randomly selected, the difference vector of the two is calculated and weighted to generate a mutated solution, and the weight is controlled by a scaling factor. Then, the mutated solution and the original solution are mixed according to the probability by crossover operation to generate a trial solution to maintain the diversity of the population. The fitness of the trial solution and the original solution is evaluated by the objective function, and only the better one is retained into the next generation population to form a selection mechanism of "survival of the fittest". The process continues until the termination condition is met, such as reaching the maximum number of iterations or the convergence threshold of the fitness, and finally the global optimal solution is output.

[0137] In the coordination layer of the application, a differential evolution algorithm is used to optimize the resource allocation scheme. In a specific implementation, the differential evolution algorithm first aggregates three types of inputs: maintenance applications submitted by bottom-layer bridge and tunnel level agents (including bridge and tunnel states, optimal and alternative maintenance schemes, and costs), strategy weight vectors issued by upper-layer network level agents, and system-level hard constraints (such as total budget). By analyzing the strategy weight, the coordination layer converts the weight strategy into a calculable priority evaluation model, generating a bridge and tunnel maintenance priority sequence. Based on this sequence, the algorithm initializes a population, with each individual corresponding to a candidate scheme of resource allocation proportions. During the evolution process, the mutation operation is as shown in formula (5):

[0138] (5)

[0139] wherein, represents a mutated individual, i.e., a new individual generated by the mutation operation, represents the index of the target individual in the current population, and represents the mutation result of the target individual, for example, if there are individuals in the population, , then is a mutated individual generated for . This generated mutated individual is generated by randomly selecting three different individuals. Here, , , , represent the three randomly selected different individuals, and the scaling factor controls the disturbance intensity, and a new scheme is generated by differential vector disturbance. The crossover operation mixes the new and old schemes with a probability of 50% to maintain diversity, and the selection operation adopts an elite preservation strategy to preserve individuals with better fitness in the parent population and the child population. If the scheme exceeds the budget, a boundary processing strategy is used to adjust the resource allocation. Finally, after multiple generations of iterative optimization, the resource allocation scheme with the best fitness is output to guide the bottom-layer execution.

[0140] The overall process of the application is based on the three-layer architecture of the network layer, the intermediate layer, and the bridge and tunnel layer. The algorithms in each layer work closely together to achieve efficient bridge and tunnel maintenance decision-making. The overall GA-RL layered multi-agent bridge and tunnel maintenance decision-making flowchart is shown in Figure 3 .

[0141] In the entire decision-making process, the algorithm requires two types of inputs: the data information of the bridges and tunnels of the road network bridge and tunnel group, and the expert knowledge input. The process is shown in Figure 4 .

[0142] ​The upper GA algorithm utilizes the expert strategy library generated by expert knowledge to select the final K strategies as the action space of the network-level agent in the network layer by TOP-K selection method. The action space of the network-level agent is the strategy optimized by the GA algorithm, the state space is the network health degree of the bridge and tunnel group network, the annual budget, the bridge to be maintained and the optimal maintenance action cost and the suboptimal maintenance action cost, and the reward function is to maximize the network health degree. The final algorithm can output the maintenance strategy under various budget constraints at any time according to the state of the bridge and tunnel group.

[0143] The present application proposes a hierarchical multi-agent bridge and tunnel maintenance decision-making method based on genetic algorithm-reinforcement learning (GA-RL), aiming to reconstruct the decision path by the complementary advantages of genetic algorithm and deep reinforcement learning to solve the following two technical problems at the same time:

[0144] Break through the calculation bottleneck of dimension explosion of state space and action space: through the hierarchical multi-agent architecture, the global strategy generation and local decision optimization are decoupled, the network-level agent pre-generates a high-quality strategy library by using the genetic algorithm, and the bridge and tunnel-level RL agent cluster focuses on local state optimization, avoiding the dimension disaster caused by centralized calculation.

[0145] Overcome the learning mechanism failure problem under rigid constraints: the coordination layer converts the hard constraints (such as total budget, risk threshold) into calculable resource allocation rules through dynamic budget allocation and priority evaluation model, combines the constraint adaptive strategy library pre-generated by GA, and realizes the collaborative optimization of strategy learning and constraint satisfaction.

[0146] The present application provides an efficient, robust and scalable solution for complex bridge and tunnel network maintenance decision-making through the innovative integration of hierarchical multi-agent architecture, GA algorithm guided by expert knowledge and constraint adaptive coordination mechanism, which significantly improves the maintenance efficiency and safety of large-scale bridge and tunnel network of road network.

[0147] The present application realizes technical breakthrough through the following five core technical features, which together constitute the core of the present application:

[0148] 1. Hierarchical multi-agent collaborative architecture design: the present application first proposes a three-level decoupling architecture of "network level-coordination layer-bridge and tunnel level", realizing the functional separation of global strategy generation and local decision optimization. The network-level agent pre-generates a constraint adaptive strategy library by GA algorithm, the bridge and tunnel-level agent cluster trains a dedicated PPO model for each type of bridge and tunnel, and the coordination layer dynamically converts rigid constraints into resource allocation problems. This architecture avoids the state space explosion problem of centralized decision-making, and at the same time ensures the consistency of decision-making through information closed loop (bridge and tunnel level state→coordination layer priority→network level strategy selection). Without this architecture, traditional methods cannot handle the real-time decision-making needs of hundreds of bridge and tunnel networks.

[0149] 2. GA-RL Hybrid Decision-Making Mechanism: This innovative approach integrates the GA algorithm with DRL, forming a dynamic collaborative mechanism of "global exploration + local fine-tuning." GA pre-generates a high-quality TOP-K policy library through expert-guided initialization, real-valued encoding (weight vector + threshold vector), and adaptive evolutionary operators (roulette wheel selection, elite retention mechanism, boundary-protected SBX crossover, and Gaussian mutation). DRL then selects the optimal policy from the discrete policy library, learning policy selection through interaction with the environment to maximize long-term rewards. This mechanism addresses the shortcomings of pure GA (lacking real-time adaptability) and pure RL (prone to local optima).

[0150] 3. Multi-agent parameter sharing and type adaptation mechanism: Addressing the diversity and significant differences in scale among bridge and tunnel types, a deployment strategy of "dedicated training + parameter sharing" is proposed. A PPO agent is trained independently for each type of bridge and tunnel (ensuring a decision adaptation degradation mechanism), while bridges and tunnels of the same type share the same model parameters, avoiding redundant modeling overhead. When a new bridge or tunnel type is added, only the corresponding type of agent needs to be trained for seamless expansion. Without this mechanism, the computational complexity of a network with thousands of bridges would increase exponentially, failing to meet real-time decision-making requirements.

[0151] 4. Rigid Constraint Adaptive Coordination Mechanism: This mechanism transforms hard constraints such as total budget and risk thresholds into computable dynamic resource allocation rules. The coordination layer generates maintenance sequences through a priority evaluation model. Based on these sequences, a differential evolution algorithm is used to optimize the budget allocation ratio, resulting in a budget allocation scheme. This mechanism addresses the problem of traditional RL algorithms struggling to handle rigid constraints.

[0152] 5. Deep Adaptation Design of Evolutionary Operators: To address the specific needs of bridge and tunnel maintenance, a dedicated design was implemented for the chromosome encoding and evolutionary operators in the GA algorithm: chromosomes employ real-valued encoding (previous...). Bit weight vector, later (bit threshold vector), ensuring the physical meaning of the parameters is interpretable; in the mutation operator, the weighted gene segments are normalized after Gaussian perturbation (satisfying...) The threshold gene segment undergoes non-uniform mutation (the amplitude decays with the number of iterations) to avoid violating common sense in engineering (such as exceeding the degradation threshold). Without this design, the general GA operator will be unable to generate strategies that conform to actual engineering constraints.

[0153] The five technical features above are interdependent and jointly solve the fundamental problems of traditional methods in state space explosion, constraint adaptation, type expansion, etc. For example, the hierarchical architecture (feature 1) provides an implementation framework for the GA-RL hybrid mechanism (feature 2), while parameter sharing (feature 3) and constraint coordination (feature 4) ensure the scalability and compliance of the decision, and the evolution operator adaptation (feature 5) further improves the strategy quality of GA. These features together constitute the core innovation point of the invention that distinguishes it from existing technologies.

[0154] In the embodiments of the present application, the computational bottleneck of dimension explosion of state space and action space is broken: traditional single-agent RL is difficult to make decisions in a hundred-bridge tunnel network due to the exponential growth of action space (N^M, where N is the number of bridges and M is the number of actions for a single bridge) , N is the number of bridges, M is the number of actions for a single bridge). The present application uses a hierarchical architecture to separate global strategy generation and local decision optimization (bridge-tunnel decoupling, computational complexity reduced to O(N+M) ). Network-level agents use pre-generated strategy libraries to compress the global action space from space N^M to the number of strategies in the strategy library .

[0155] Adaptive policy learning under rigid constraints: Traditional penalty function method is prone to conservative or non-compliant strategies due to difficulty in setting penalty coefficients. The present application converts the total budget into a resource allocation optimization problem through the coordination layer, and uses a differential evolution algorithm for resource allocation to ensure that the final decision scheme strictly complies with the constraints.

[0156] Flexible expansion of bridge and tunnel types and sizes: The present application uses the mechanism of dedicated type training of bottom-level agent clusters and parameter sharing deployment to build a fully scalable decision framework, achieving flexible expansion of bridge and tunnel types and sizes. In the type dimension, dedicated PPO agents are trained independently for different types of bridges and tunnels such as beam bridges, arch bridges, and cable-stayed bridges to ensure accurate adaptation of the decision to the deterioration mechanism of each type of bridge and tunnel. When a new type (such as a rigid-frame bridge) is added, only the corresponding agent needs to be trained to seamlessly expand. In the quantity dimension, the same model parameters are shared by bridges and tunnels of the same type through the parameter sharing mechanism, avoiding the need to create independent models for each bridge and tunnel, and reducing the consumption of computing resources.

[0157] Figure 5 is a GA-RL-based hierarchical multi-agent bridge and tunnel group maintenance decision device diagram according to an exemplary embodiment, which is used for a GA-RL-based hierarchical multi-agent bridge and tunnel group maintenance decision method. Referring to Figure 5 , the device includes a data acquisition module 310, a network layer module 320, a bridge and tunnel layer module 330, and an intermediate layer module 340. Among them:

[0158] ​​The data acquisition module 310 is configured to acquire data information of the bridge and tunnel group and an expert strategy.

[0159] The network layer module 320 is configured to construct an expert strategy library offline according to the expert strategy and a genetic algorithm, and select an optimal strategy online according to the expert strategy library and a network-level reinforcement learning agent.

[0160] The bridge and tunnel layer module 330 is configured to train a dedicated bridge and tunnel-level reinforcement learning agent for each bridge and tunnel type by using a proximal policy optimization algorithm, construct a bridge and tunnel agent cluster, and generate a maintenance application for a bridge and tunnel to be maintained by the bridge and tunnel-level reinforcement learning agent.

[0161] The intermediate layer module 340 is configured to convert the optimal strategy into a calculable priority evaluation model according to the optimal strategy, the maintenance application and the acquired data information, generate a bridge and tunnel maintenance priority sequence according to the priority evaluation model, optimize a budget allocation ratio according to the bridge and tunnel maintenance priority sequence and a differential evolution algorithm, generate a bridge and tunnel group network-level maintenance scheme, and execute the maintenance scheme by the bridge and tunnel-level reinforcement learning agent.

[0162] In the embodiment of the application, the calculation bottleneck of dimension explosion of state space and action space is broken through: the traditional single-agent RL is difficult to make decisions in a hundred-bridge and tunnel network due to the exponential growth of the action space (N , is the number of single-bridge actions, and N is the number of bridges). The global strategy generation and local decision optimization are decoupled by the hierarchical architecture, and the calculation complexity is reduced to O (N ). The global action space is compressed from a space of N to the number of strategies in the strategy library .

[0163] Adaptive policy learning under rigid constraints: the traditional penalty function method is prone to conservative or default strategies due to the difficulty in setting the penalty coefficient. The total budget is converted into a resource allocation optimization problem by the coordination layer, and the differential evolution algorithm is used for resource allocation to ensure that the final decision scheme strictly complies with the constraints.

[0164] Flexible expansion capability of bridge and tunnel types and scales: the application constructs a full-dimension expandable decision framework by the dedicated type training of the bottom-layer agent cluster and the parameter sharing deployment mechanism, and realizes the flexible expansion capability of bridge and tunnel types and scales. In the type dimension, the dedicated PPO agent is independently trained for the differentiated bridge and tunnel types such as beam bridges, arch bridges and cable-stayed bridges, to ensure that the decision accurately adapts to the deterioration mechanism of various bridge and tunnels; when a new type (such as a rigid frame bridge) is added, only the corresponding agent needs to be trained to seamlessly expand. In the number dimension, the same model parameters are shared by the same type of bridge and tunnel through the parameter sharing mechanism, avoiding the repeated creation of independent models for each bridge and tunnel, and reducing the consumption of computing resources.

[0165] Figure 6 This is a schematic diagram of the structure of a hierarchical multi-agent large-scale bridge and tunnel group maintenance decision-making device provided in an embodiment of the present invention, as shown below. Figure 6 As shown, a hierarchical multi-agent large-scale bridge and tunnel group maintenance decision-making device may include the above-mentioned Figure 5 The illustrated hierarchical multi-agent bridge and tunnel group maintenance decision-making device is based on GA-RL. Optionally, the hierarchical multi-agent large-scale bridge and tunnel group maintenance decision-making device 410 may include a first processor 2001.

[0166] Optionally, the hierarchical multi-agent large-scale bridge and tunnel group maintenance decision-making device 410 may also include a memory 2002 and a transceiver 2003.

[0167] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0168] The following is combined with Figure 6 A detailed description of each component of the hierarchical multi-agent large-scale bridge and tunnel group maintenance decision-making device 410 is provided below:

[0169] The first processor 2001 is the control center of the hierarchical multi-agent large-scale bridge and tunnel group maintenance decision-making device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0170] Optionally, the first processor 2001 can perform various functions of the hierarchical multi-agent large-scale bridge and tunnel group maintenance decision-making device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0171] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 6 CPU0 and CPU1 are shown in the diagram.

[0172] In a specific implementation, as an embodiment, the layered multi-agent large-scale bridge-tunnel group maintenance decision device 410 can also include a plurality of processors, for example, the first processor 2001 and the second processor 2004 shown in FIG. 2. Figure 6 Each of the processors can be a single-CPU or a multi-CPU. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (for example, computer program instructions).

[0173] The memory 2002 is configured to store a software program for implementing the scheme of the present application, and the first processor 2001 is configured to control execution. For specific implementation, refer to the above-mentioned method embodiments, which will not be repeated here.

[0174] Optionally, the memory 2002 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, an optical disk storage (including a compact disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but not limited to this. The memory 2002 can be integrated with the first processor 2001 or exist independently and be coupled to the first processor 2001 through an interface circuit (not shown in FIG. 2) of the large-scale layered multi-agent bridge-tunnel group maintenance decision device 410, and the embodiments of the present application are not limited in this regard. Figure 6

[0175] The transceiver 2003 is configured to communicate with a network device or a terminal device.

[0176] Optionally, the transceiver 2003 can include a receiver and a transmitter (not shown separately in FIG. 2). The receiver is configured to implement a receiving function, and the transmitter is configured to implement a transmitting function. Figure 6

[0177] ​​Optionally, the transceiver 2003 can be integrated with the first processor 2001, or exist independently, and is coupled with the first processor 2001 through an interface circuit (not shown in the figure) of the large-scale layered multi-agent bridge and tunnel group maintenance decision device 410, and the embodiments of the present application are not limited thereto. Figure 6 The first processor 2001 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc.

[0178] It should be noted that, Figure 6 The structure of the layered multi-agent large-scale bridge and tunnel group maintenance decision device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure identification device can include more or fewer components than those shown in the figure, or combine certain components, or different component arrangements.

[0179] In addition, the technical effects of the layered multi-agent large-scale bridge and tunnel group maintenance decision device 410 can refer to the technical effects of the layered multi-agent large-scale bridge and tunnel group maintenance decision method based on GA-RL described above, which will not be repeated here.

[0180] It should be understood that the first processor 2001 in the embodiments of the present application can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0181] It should also be understood that the memory in the embodiments of the present application can be volatile or nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. The volatile memory can be random access memory (RAM) used as external cache. By way of example, and not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0182] The above-described embodiments can be implemented in whole or in part by software, hardware (such as a circuit), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0183] It should be understood that the term "and / or" herein merely describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents that the associated objects before and after it are in an "or" relationship, but it can also represent an "and / or" relationship, which can be understood according to the context before and after it.

[0184] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of the items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0185] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined according to their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0186] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0187] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0188] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0189] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0190] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0191] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0192] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A hierarchical multi-agent bridge and tunnel group maintenance decision-making method based on GA-RL, characterized in that, The method includes: S1. Obtain data and expert strategies for the road network, bridges, and tunnels; S2. Construct an expert policy library offline based on expert policies and genetic algorithms, and select the optimal policy online based on the expert policy library and the network-level reinforcement learning agent. S3. Use the near-end policy optimization algorithm to train a dedicated bridge and tunnel level reinforcement learning agent for each type of bridge and tunnel, build a bridge and tunnel agent cluster, and generate maintenance requests for bridges and tunnels to be repaired through the bridge and tunnel level reinforcement learning agent. S4. Based on the optimal strategy, maintenance application and acquired data information, generate a bridge and tunnel maintenance priority sequence according to the priority evaluation model, optimize the budget allocation ratio according to the bridge and tunnel maintenance priority sequence and differential evolution algorithm, generate a bridge and tunnel group network-level maintenance plan, and execute the maintenance plan through a bridge and tunnel-level reinforcement learning agent. The step S2, which involves offline construction of an expert policy library based on expert policies and genetic algorithms, and online selection of the optimal policy based on the expert policy library and the network-level reinforcement learning agent, includes: S21. The expert strategy is formalized into a strategy vector, as shown in equation (1) below: （1） In the formula, Represents the policy vector. Represents the weight vector. This indicates the number of preferences in the expert strategy. Represents the threshold vector. This indicates the number of threshold constraints in the expert strategy; S22. Using a fixed-length vector with real-valued encoding, the strategy vector is mapped to the gene locus to obtain the chromosome, as shown in equation (2) below: （2） In the formula, Indicates chromosome; S23. Generate the initial population based on chromosomes and Latin hypercube sampling; S24. Design an evolutionary operator to generate a new strategy by performing a global exploration in the solution space based on the initial population and the evolutionary operator. S25. Select the top-K new strategies with the best performance and build an expert strategy library offline; where K is a preset value. S26. Construct an action space based on the expert policy and TOP-K new policies, and select the optimal policy online from the action space through a network-level reinforcement learning agent; The maintenance request for the bridge and tunnel to be repaired generated by the bridge and tunnel-level reinforcement learning agent in S3 includes: The bridge and tunnel level reinforcement learning agent calculates the probability distribution of all feasible maintenance actions under the current state of the bridge and tunnel to be repaired, and takes the maintenance action with the highest probability value as the optimal maintenance action suggestion, and the maintenance action with the second highest probability value as the alternative maintenance action suggestion. A maintenance request is generated based on the optimal maintenance action suggestion and the alternative maintenance action suggestion.

2. The hierarchical multi-agent bridge and tunnel group maintenance decision-making method based on GA-RL according to claim 1, characterized in that, The selection operator of the evolutionary operator in S24 uses network health as the fitness of the genetic algorithm and combines roulette wheel selection and elite retention strategies. The crossover operator introduces boundary protection to simulate binary crossover; The mutation operator employs Gaussian perturbation and non-uniform mutation.

3. The hierarchical multi-agent bridge and tunnel group maintenance decision-making method based on GA-RL according to claim 1, characterized in that, The state space of the network-level reinforcement learning agent in S26 is: the network health of the bridge and tunnel agent cluster, the annual maintenance budget, the bridge and tunnel to be repaired, and the optimal maintenance action suggestion cost and alternative maintenance action suggestion cost of the bridge and tunnel to be repaired. The reward function for a network-level reinforcement learning agent is to maximize network health.

4. The hierarchical multi-agent bridge and tunnel group maintenance decision-making method based on GA-RL according to claim 1, characterized in that, The S3 algorithm employs a proximal policy optimization algorithm to train a dedicated bridge-tunnel level reinforcement learning agent for each bridge-tunnel type, constructing a bridge-tunnel agent cluster, including: A proximal policy optimization algorithm is used to train a dedicated bridge-tunnel level reinforcement learning agent for each bridge-tunnel type. The bridge-tunnel level reinforcement learning agents of multiple bridge-tunnel types are integrated to obtain a bridge-tunnel agent cluster. Among them, all different bridge-tunnel instances serving the same bridge-tunnel type share the same trained bridge-tunnel level reinforcement learning agent and parameters.

5. The hierarchical multi-agent bridge and tunnel group maintenance decision-making method based on GA-RL according to claim 1, characterized in that, The parameter update follows the principle of collective optimization, as shown in equation (3) below: （3） In the formula, This indicates the strategy network parameters corresponding to the bridge / tunnel type. Represented as the learning rate parameter, Indicates the number of bridge and tunnel types. To sum the gradients of all bridge and tunnel instances of the same type, representing the collective optimization direction, These represent the parameters of the policy network.

6. The hierarchical multi-agent bridge and tunnel group maintenance decision-making method based on GA-RL according to claim 1, characterized in that, The mutation operation in the differential evolution algorithm in S4 is shown in equation (4) below: （4） In the formula, Indicates a variant individual. Indicates the index of the target individual in the population. This represents three randomly selected distinct individuals. This represents the scaling factor.

7. A hierarchical multi-agent bridge and tunnel group maintenance decision-making device based on GA-RL, wherein the hierarchical multi-agent bridge and tunnel group maintenance decision-making device based on GA-RL is used to implement the hierarchical multi-agent bridge and tunnel group maintenance decision-making method based on GA-RL as described in any one of claims 1-6, characterized in that, The device includes: The data acquisition module is used to acquire data information and expert strategies for road network bridge and tunnel groups; The network layer module is used to build an expert policy library offline based on expert policies and genetic algorithms, and to select the optimal policy online based on the expert policy library and the network-level reinforcement learning agent. The bridge and tunnel layer module is used to train a dedicated bridge and tunnel-level reinforcement learning agent for each type of bridge and tunnel using a near-end policy optimization algorithm, build a bridge and tunnel agent cluster, and generate maintenance requests for bridges and tunnels to be repaired through the bridge and tunnel-level reinforcement learning agent. The intermediate layer module is used to transform the optimal strategy into a computable priority evaluation model based on the optimal strategy, maintenance application and acquired data information, generate a bridge and tunnel maintenance priority sequence based on the priority evaluation model, optimize the budget allocation ratio based on the bridge and tunnel maintenance priority sequence and differential evolution algorithm, generate a bridge and tunnel group network-level maintenance plan, and execute the maintenance plan through a bridge and tunnel-level reinforcement learning agent.

8. A hierarchical multi-agent large-scale bridge and tunnel group maintenance decision-making device, characterized in that, The hierarchical multi-agent large-scale bridge and tunnel group maintenance decision-making equipment includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images