A highway overload management collaborative decision and disposal method based on multi-agent reinforcement learning

Abstract

The application discloses a highway overweight management collaborative decision and disposal method based on multi-agent reinforcement learning. In view of technical problems in highway overweight management, such as difficulty in credit distribution of multiple departments, low strategy coordination efficiency under a partially observable environment, slow learning caused by sparse rewards and the like, a bidirectional attention credit distribution algorithm is designed, non-monotonic credit distribution is realized through a positive and negative credit flow separation mechanism, a graph attention collaborative decision algorithm is adopted, a department relationship graph is dynamically constructed to realize directional communication and reduce communication overhead, a double-layer adaptive internal reward algorithm is constructed, and internal reward symbols are bound with credit gradients to guide beneficial exploration, and the algorithm is interlocked and coupled through a joint loss function and is cooperatively optimized. The application realizes full-process automatic collaborative decision from state perception to disposal execution, and significantly improves response speed and strategy effect of overweight management.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and multi-agent systems technology, specifically relating to a collaborative decision-making and handling method for highway overload management based on multi-agent reinforcement learning. Background Technology

[0002] The existing technology mainly faces the following technical challenges:

[0003] Multi-departmental collaborative decision-making is difficult: Overweight governance involves multiple departments, each with different objective functions. Existing solutions employ a manual collaboration model of joint meetings and information sharing, resulting in decision-making response delays of several hours to days, failing to achieve real-time collaboration.

[0004] Credit allocation is challenging because the effectiveness of governance is a collective contribution, making it difficult to quantify the decision-making contributions of individual departments. Traditional multi-agent reinforcement learning methods, such as QMIX, achieve credit allocation through monotonic hybrid networks, but they assume all agents contribute in the same direction, failing to handle scenarios with incompatible interests. While recent methods like XMIX introduce graph neural networks to improve credit allocation, they are still limited by the monotonicity assumption, making it difficult to handle conflicts of interest among heterogeneous agents.

[0005] Coordinating strategies under partially observable conditions is challenging: each department only possesses local information and cannot obtain the global state. Existing graph attention-based methods, while able to model agent interactions through graph structures, still suffer from significant communication overhead as the number of departments increases, and do not consider the dynamic differences in the importance of communication content.

[0006] The simplistic approach leads to policy degradation: the existing system only handles overweight vehicles by unloading and imposing fines, without a dynamic reward and punishment mechanism. In the reinforcement learning framework, sparse external rewards result in slow policy convergence. Summary of the Invention

[0007] This invention aims to solve the technical problems in existing highway overload management, such as difficulties in credit allocation, low efficiency of strategy coordination in some observable environments, and low learning efficiency caused by sparse rewards, and provides a collaborative decision-making and handling method based on multi-agent reinforcement learning.

[0008] This invention employs a centralized training and distributed execution architecture, designing nonlinear algorithm modules to work collaboratively, forming a hierarchically coupled, mutually inputting, and jointly optimized organic whole: the bottom-level credit allocation module, i.e., Algorithm 1, is responsible for solving the problem of who contributed value; the middle-level collaborative communication module, i.e., Algorithm 2, is responsible for solving the problem of who needs to communicate with whom; and the top-level incentive shaping module, i.e., Algorithm 3, is responsible for solving the problem of what behavior deserves a reward; through a unified loss function, interlocking coupling at the gradient level is achieved, and parameter updates of any algorithm will affect the optimization direction of the other two algorithms.

[0009] Algorithm 1: Multi-sectoral heterogeneous credit allocation algorithm based on bidirectional attention mechanism

[0010] The existing QMIX algorithm assumes a monotonic relationship between individual action value and joint action value, but it fails to address situations where individual decisions do not consider the common interests of multiple parties. Recent research indicates that under conditions of shared global rewards, the strategies of multiple agents tend to homogenize due to similar observations, leading to ambiguous reward attribution. This algorithm introduces a mechanism to separate positive and negative credit flows, using a gradient-driven contribution quantification method to distinguish the heterogeneous impact of each department on the global reward.

[0011] The decision contribution of the i-th department, i.e., the agent, is decomposed as follows:

[0012]

[0013] in, The value function representing joint actions indicates the long-term expected return of multi-department joint decision-making. Historical observations and action trajectories representing various departments , where N is the total number of departments; The vector representing the joint action of each department. ; The function representing the value of individual actions in the i-th department; Weighting coefficients with values ​​between 0 and 1 represent positive contribution weights; The weight coefficient, ranging from 0 to 1, represents the negative suppression weight, and is applied when negative credit flows are activated. =0, The suppression strength is determined independently, and the condition is no longer mandatory. + =1; This is a positive credit flow mapper; it is responsible for mapping the weighted sum of positive individual values ​​to positive joint value components. Its design must ensure that the output monotonically increases with the input, while introducing nonlinearity to fit complex value relationships. It is a negative credit flow mapper; responsible for mapping the weighted sum of negative individual values ​​to the negative value components that need to be deducted.

[0014] Positive credit weight Dynamically calculated using a bidirectional attention mechanism:

[0015]

[0016] in, It is a normalized exponential function; This is a learnable query weight matrix; The key weight matrix is ​​a learnable matrix. Dimensions for each attention head; For the credit gating function, if If the value is less than 0, output 0; otherwise, output 1, which is used to force the positive credit flow of negative contribution departments to be closed. Let i be the value of the individual action in the i-th department; For the value of joint operations, The difference between collective value and individual value. ,when When the value is less than 0, the department is considered to have made a negative contribution, and its positive weight is determined accordingly. It is forcibly set to zero, while simultaneously activating the negative credit flow branch; This represents the matrix transpose operation.

[0017] The update of negative suppression weights is driven by the credit gradient. This addresses the problem of traditional methods struggling to distinguish the contributions of heterogeneous individuals. Update according to the following rules:

[0018]

[0019] in, Let be the negative suppression weight for the i-th department; To negatively suppress the learning rate of weight updates, the step size of each update is controlled; Indicates to Find the gradient; The credit gradient measures the marginal contribution of an individual's value to the collective value. This mechanism borrows from gradient-based contribution quantification strategies, enabling negative inhibition weights to adaptively respond to the actual direction of an individual's behavior's impact on the collective goal.

[0020] The nonlinearity and non-monotonicity of the algorithm are reflected in the following aspects: the credit flow separation operation breaks the monotonicity constraint of QMIX and can model non-monotonic tasks such as alternating collaboration; the Softmax attention mechanism introduces nonlinear interaction relationships between departments; the Mask gating realizes discontinuous piecewise mapping; the gradient-driven weight update process introduces a nonlinear feedback loop between individuals and the collective.

[0021] Algorithm 2: Partially Observable Collaborative Decision-Making Algorithm Based on Graph Attention Mechanism

[0022] In partially observable environments, agents typically only acquire local observations, leading to inefficient exploration of cooperative strategies. This algorithm constructs a dynamic departmental relationship graph and uses a graph attention network to achieve targeted propagation of important information, avoiding the communication explosion problem caused by full broadcasting. Graph structures are naturally adapted to multi-departmental systems—modeling each department as a node in the graph and the collaborative relationships between departments as edges, thus transforming the multi-departmental collaboration problem into an information transmission and aggregation problem on the graph. The key advantage of the graph attention mechanism compared to fixed topology methods is that when neighboring nodes have similar characteristics, the fixed topology assumption cannot adapt to dynamic changes in the environment, while the attention mechanism can adaptively adjust the correlation weights between nodes.

[0023] Define the departmental relationship diagram at time t as follows: Where the node set V represents N departments, and the edge set V represents the number of departments. Attention coefficient The decision is made. To ensure that the convolution operation is independent of the order of the input feature vectors, i.e., to satisfy permutation invariance, a multi-head dot product attention mechanism is adopted. The attention correlation score between any two departments i and j is determined. The calculation is as follows:

[0024]

[0025] in, The activation function is a linear rectifier with leakage, and the slope in the negative interval is 0.2. Let be the attention-related score between department i and department j at time t; This is a learnable attention parameter vector; the superscript T indicates vector transpose. For shared linear transformation matrices; The hidden state of the i-th department at time t is obtained by encoding its local observations through a gated cyclic unit; Let be the hidden state of the j-th department at time t; This indicates a vector concatenation operation.

[0026] The scores above are normalized to obtain the final attention coefficient:

[0027]

[0028] in, The normalized attention coefficient represents the attention weight of department i to department j; It is an exponential function with the natural constant e as its base; Attention-related scores; Let be the set of neighboring nodes of department i; the denominator represents the sum of the relevant scores of all neighboring nodes of department i.

[0029] To enhance representation capabilities, a multi-head attention mechanism is employed. Each attention head independently calculates the correlation weights between departments, and the updates for each department are based on the aggregated neighbor information.

[0030]

[0031] in, This represents the hidden state of the i-th department after aggregation and updating. Let be the attention coefficient of department i to department j in the m-th attention head; Let be the hidden state of the j-th department at time t; For the total number of attention heads; The transformation matrix for the value of the m-th attention head; (·) represents a non-linear activation function, using the ReLU function; This means concatenating the outputs of multiple attention heads along the feature dimension.

[0032] By stacking multiple graph attention layers, the agent's receptive field gradually expands, enabling it to capture higher-order interdepartmental relationships and thus acquire a wider range of collaborative information without increasing communication overhead.

[0033] The action selection strategy is defined as follows:

[0034]

[0035] in, Let i be the policy function for the i-th department, and let i be the probability distribution of choosing each action given the historical trajectory and the hidden state. Let i be the action performed by the i-th department at time t; The historical observation-action trajectory of the i-th department up to time t; This is the hidden state before the update; The updated hidden state; MLP is a multilayer perceptron, whose output dimension is equal to the action space size of the i-th department.

[0036] The algorithm's advantages in nonlinearity and complexity are reflected in the following aspects: the calculation of attention coefficients involves LeakyReLU nonlinear activation; Softmax normalization implements a nonlinear probability distribution on neighboring nodes; the splicing operation of multi-head attention introduces nonlinear fusion of multiple representation spaces; and the communication complexity is reduced from O(N) through the attention mechanism. 2 The bandwidth can be reduced to O(N·K), where K is the average number of neighbors and is much smaller than N, which can save more than 80% of the communication bandwidth compared to the full broadcast scheme.

[0037] Algorithm 3: Two-layer adaptive intrinsic reward generation algorithm

[0038] To address the problem of sparse external rewards in highway overload management—where positive rewards are only awarded when overloaded vehicles are successfully apprehended—this algorithm designs an intrinsic reward generation network, enabling agents to obtain continuous and dense shaping rewards during the exploration phase. Traditional intrinsic reward methods, such as random network distillation, can generate continuous reward signals, but their intrinsic rewards lack direct coupling with the collective goal, potentially leading to deviations between individual behavior and team objectives in multi-agent collaborative scenarios.

[0039] Define the total reward obtained by the i-th agent at time t as:

[0040]

[0041] in, Let be the total reward received by the i-th department at time t; This signal represents external rewards received from the environment and is relatively sparse. Represents the intrinsic reward function, Let be the global state at time t. The action performed by the i-th department at time t must satisfy the consistency constraint with the collective goal; This is an adaptive adjustment coefficient that gradually decreases as the training process progresses.

[0042] Intrinsic reward function Generated by a learnable network, its optimization objective is to maximize the positive correlation with the collective value function. Borrowing from gradient-based optimization of intrinsic rewards, it correlates intrinsic rewards with credit sensitivity:

[0043]

[0044] in, (·) is a parameter The learnable multilayer perceptron outputs a scalar form of reward intensity; sign(·) is the sign function, ensuring that the positive or negative direction of the intrinsic reward is consistent with the sensitivity of collective value to individual value; Inheriting the credit gradient from Algorithm 1, it achieves end-to-end coupling between credit allocation and reward shaping.

[0045] The key innovation of this design lies in the fact that the symbol of intrinsic reward—whether it is a positive incentive or a negative treatment—is not pre-set by external rules, but dynamically determined by the direction of an individual's marginal contribution to the collective goal. When a department's decision makes a positive marginal contribution to the joint value, the symbol function value is positive, and the intrinsic reward further reinforces the behavior; conversely, it generates a negative existential cue, inhibiting the behavior.

[0046] Adaptive coefficients A two-level update rule is adopted:

[0047] Inner layer, i.e., individual-level updates: Each department maintains a learning progress counter. :

[0048]

[0049] in, Let be the learning progress counter for the i-th department at time t; This is the learning progress counter value from the previous moment; (·) is an indicator function that takes the value 1 when the condition inside the parentheses is true, and takes the value 0 otherwise. Let i be the value of the action of the i-th department at the current moment; Let i be the value of the action of the i-th department at the previous moment; The threshold for value change is used; this counter records the number of times an individual's value function changes significantly, serving as a proxy metric for learning progress.

[0050] Outer layer, i.e., collective-level update: Employs an exponential scheduling strategy based on collective performance improvement rate.

[0051]

[0052] in, This is the adaptive decay coefficient for the next time step; This represents the adaptive decay coefficient at the current moment; This represents the improvement in collective performance over the most recent t training rounds. This represents the collective performance index at the current moment. This serves as the historical average performance baseline. This is the attenuation rate hyperparameter.

[0053] The core logic of the two-layer structure is as follows: when collective performance improves rapidly, it indicates that external rewards are effectively guiding learning, and the weight of intrinsic rewards should be reduced to allow the agent to focus on the real task objective; when collective performance stagnates, the decay rate slows down or even rebounds, and intrinsic rewards continue to play a guiding role in exploration. The inner layer's individual progress counters provide fine-grained reference information for possible multi-scale scheduling.

[0054] The nonlinearity and two-layer characteristics of this algorithm are reflected in the following aspects: the intrinsic reward generation network MLP introduces parameterized nonlinear mapping; the sign function realizes symbolic discontinuous decision-making; the two-layer optimization structure couples individual learning progress with collective scheduling; and the multiplicative coupling of intrinsic reward and credit gradient forms a nonlinear feedback loop.

[0055] Collaborative relationships between algorithms: Algorithms form a hierarchically coupled, mutually inputting, and jointly optimizing organic whole. Their logical relationships can be characterized from the following dimensions:

[0056] The hierarchical relationship of the objectives is as follows: A top-down decomposition chain and a bottom-up aggregation chain are formed from the collective level to the credit level. At the collective level (top layer), Algorithm 3 is responsible for solving the sparse reward problem. Its inputs are external environmental rewards, collective performance indicators, and the credit gradient from Algorithm 1; its outputs are intrinsic rewards and adaptive coefficients. At the coordination level (middle layer), Algorithm 2 is responsible for solving the information sharing problem under partially observable conditions. Its inputs are local observations from each department and aggregated neighbor embeddings; its outputs are enhanced state representations and policy distributions. At the credit level (bottom layer), Algorithm 1 is responsible for solving the credit allocation problem between individual contributions and collective objectives. Its inputs are individual action values ​​and features enhanced by Algorithm 2; its outputs are credit weights, joint action values, and credit gradients.

[0057] The top-down decomposition chain is as follows: Algorithm 3 determines the types of behaviors worthy of reward, guides the exploration direction through intrinsic rewards, and its reward sign depends on the credit gradient of Algorithm 1; Algorithm 2 determines the departmental objects that need to communicate, determines the information aggregation objects through graph attention, and its policy gradient depends on the total reward generated by Algorithm 3; Algorithm 1 determines the contribution sources that should be amplified or suppressed, determines the value aggregation weights through credit flow, and its input features depend on the enhanced representation output by Algorithm 2.

[0058] The bottom-up aggregation chain is as follows: the credit gradient output by Algorithm 1 serves as the basis for determining the intrinsic reward sign in Algorithm 3; the enhanced state output by Algorithm 2 serves as the input feature for calculating the individual value function in Algorithm 1; and the total reward output by Algorithm 3 serves as the reward signal for calculating the policy gradient in Algorithm 2.

[0059] Data dependency and coupling. During the training phase, the data flow path between the algorithms is as follows: the external environment reward first enters Algorithm 3, while the credit gradient flows from Algorithm 1 to Algorithm 3; after generating the total reward, Algorithm 3 feeds it into the policy network of each agent; after the policy network executes the action, it passes the local observation to Algorithm 2; after Algorithm 2 completes graph attention aggregation, it passes the enhanced hidden state to Algorithm 1; Algorithm 1 performs credit allocation calculation and outputs the joint action value; finally, it calculates the temporal difference error and simultaneously feeds the gradient back to the algorithm module.

[0060] Gradient coupling. The algorithm achieves interlocking coupling at the gradient level through a unified loss function. Total loss function. It consists of a weighted average of four components: temporal difference loss, credit allocation loss, collaborative decision-making loss, and intrinsic reward generation loss, defined as follows:

[0061]

[0062] in, The master temporal difference loss is used to update the global joint action value function. , is the core monitoring signal for model convergence; The credit allocation loss originates from the positive credit flow mapper f in Algorithm 1. pos With negative credit flow mapper f neg The prediction error is used to constrain the reasonable distribution of credit weights; The collaborative decision loss is derived from the cross-entropy between the enhanced features output by the graph attention module in Algorithm 2 and the target policy, and is used to regularize the graph communication structure. The loss generated for intrinsic reward is derived from the intrinsic reward network in Algorithm 3. The fitting error between the output and the credit gradient sign ensures that the intrinsic reward direction is consistent with the collective objective. A loss weight of 0.3 is assigned to credit; The weight for the collaborative decision-making loss is set to 0.2. A loss weight is generated for the intrinsic reward, with a value of 0.2.

[0063] The above weight configuration ensures that the main temporal difference loss dominates, while the three auxiliary losses—credit allocation, collaborative decision-making, and incentive shaping—participate in joint optimization with appropriate strength, avoiding any module from dominating the gradient direction and causing training imbalance.

[0064] Gradient coupling relationship: credit allocation loss The computation depends on the enhanced features output by Algorithm 2; collaborative decision loss. The policy gradient depends on the total reward generated by Algorithm 3; the intrinsic reward generation loss... The sign constraint depends on the credit gradient output by Algorithm 1. The three form an interlocked joint optimization mechanism—the parameter update of any module will affect the optimization direction of the other two modules, ensuring that the three will not converge to local optima and sacrifice global performance, but will form a dynamic equilibrium of cooperative evolution.

[0065] Gradient backpropagation process: During the centralized training phase, batch data is sampled from the experience replay pool, and forward computation is performed to obtain... Then, backpropagation simultaneously updates the credit network of Algorithm 1, the graph attention network of Algorithm 2, and the intrinsic reward generation network of Algorithm 3.

[0066] Policy networks and gating loop unit encoders for each agent.

[0067] The method of the present invention undergoes the following steps during execution:

[0068] Phase 1: Initialization and Observation Encoding. At any time t, each agent acquires its local observation data from the environment, including overload detection data, traffic flow, road conditions, and the status of its own resources. Each agent fuses the current observation with the historical hidden state through a gated cyclic encoder to generate the current hidden state representation.

[0069] Phase Two: Graph Attention-Driven Information Aggregation, or Coordination Layer Processing. Using the current hidden state as the node feature, Algorithm Two calculates the attention coefficients between departments. Each agent aggregates highly correlated information from neighboring departments to generate an enhanced hidden state. A key decision is made during this process: if the total attention coefficient of a department is lower than a preset connection threshold, that department enters an isolated mode, relying solely on its own observations for decision-making. This mechanism is used for system fault tolerance in the event of communication failures.

[0070] Phase 3: Reward Shaping, i.e., Incentive Layer Processing. The enhanced hidden state is input into the policy network, and the actions of each department are sampled. After the action is executed, the environment returns an external sparse reward. Simultaneously, Algorithm 3 calculates the intrinsic reward: an inner counter records the learning progress of each department, and the outer scheduler updates the decay coefficient based on the collective performance index. The final total reward is obtained by adding the external reward to the intrinsic reward weighted by the decay coefficient, serving as the immediate reward for this state transition.

[0071] Phase Four: Credit Allocation, or Credit Layer Processing. The enhanced hidden states, executed actions, total rewards, and historical states of each department are stored in the experience replay pool. Batch data is sampled from the replay pool, and Algorithm One calculates the individual action value, positive credit weight, and negative suppression weight for each agent, thereby obtaining the joint action value. During this process, a key judgment is made: if the difference between a department's individual value and its joint value is negative, meaning its decision failed to consider the common interests of multiple parties, then a negative credit flow is activated, and the department's contribution is suppressed in the joint value calculation.

[0072] Phase 5: Joint Gradient Update. The total loss function is calculated, and gradient descent optimization is performed on the algorithm's network parameters. Every fixed number of steps, the parameters of the policy network are copied to the target network using a soft update method to stabilize the training process.

[0073] Phase Six: Distributed Execution and Deployment. After model training converges, only the policy networks of each agent are retained during actual deployment, eliminating the need for aggregation calculations in Algorithm Two and reward generation in Algorithm Three. Each agent independently makes action decisions based on its own hidden state, with single inference latency controlled within fifty milliseconds.

[0074] The beneficial effects of this invention are as follows:

[0075] Credit allocation that overcomes monotonicity constraints. Algorithm 1, through a bidirectional credit flow separation mechanism, can identify and suppress the impact of negatively contributing sectors, solving the inherent problem of incompatible interests that existing technologies such as QMIX cannot handle. In overloaded governance scenarios, this means that when the inspection rate decreases, its negative contribution can be effectively identified and suppressed.

[0076] Reduce communication overhead. Algorithm 2 achieves dynamic directional communication through a graph attention mechanism. Compared to existing full-broadcast schemes such as CommNet, the communication complexity is reduced from quadratic to linear, saving more than 80% of communication bandwidth in scenarios involving collaboration among more than ten departments. The multi-hop propagation mechanism enables departments to obtain global situational awareness without direct communication.

[0077] Addressing the sparse reward problem, Algorithm 3 employs a two-layer adaptive intrinsic reward mechanism to provide the agent with continuous and dense learning signals. Furthermore, the intrinsic reward symbol is bound to the credit gradient, ensuring that the exploration direction remains consistent with the collective goal. Compared to existing schemes that rely solely on external sparse rewards, the policy convergence speed can be improved by more than 50%.

[0078] Module Collaborative Enhancement. This invention is the first to achieve a joint solution to three technical challenges in the context of highway overload management: credit allocation, collaborative communication, and incentive shaping. The algorithms are mutually dependent and mutually reinforcing—Algorithm 1 provides credit sensitivity to guide the reward symbol of Algorithm 3, Algorithm 2 provides enhancing features to improve the credit assessment accuracy of Algorithm 1, and Algorithm 3 provides dense rewards to accelerate the policy convergence of Algorithm 2, forming a core architecture that differs from existing technologies.

[0079] End-to-end collaborative decision-making. This invention models the multi-departmental collaborative problem of highway overweight vehicle management as a multi-agent reinforcement learning framework, realizing full-process automation from state perception to disposal execution, reducing decision latency from minutes to seconds. Attached Figure Description

[0080] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0081] Figure 1 This is a diagram of the overall system architecture of the present invention. Detailed Implementation

[0082] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0083] Algorithm novelty description:

[0084] Through retrieval and comparative analysis, the algorithm of the present invention differs from the prior art in the following aspects.

[0085] For Algorithm 1, namely the bidirectional attention credit allocation algorithm, compared with the existing QMIX and XMIX technologies, this invention has the following distinguishing features: the existing QMIX adopts a monotonic mixing method, and XMIX introduces a hybrid structure of graph neural network and attention but does not explicitly separate negative contributions, while this invention adopts a mechanism of bidirectional separation of positive and negative credit flows; existing schemes do not set explicit negative suppression branches, while this invention achieves the suppression of bad behavior through independent negative credit flow branches; the Mask gating in this invention realizes the discontinuous switching between credit flows, which is a fundamental breakthrough of the existing monotonic mixing methods.

[0086] For Algorithm 2, namely the graph attention collaborative decision-making algorithm, compared with the existing DGN and graph attention grouping multi-agent reinforcement learning methods, this invention has the following distinguishing features: the graph construction method of existing methods is based on distance or predefined rules, or based on grouping strategies, while this invention is based on dynamic attention threshold for adaptive graph construction; existing methods do not support island mode, while this invention can automatically switch to island mode when the sum of attention coefficients is lower than the threshold; multi-head attention is used for multi-representation space learning in all methods, but this invention further correlates the attention coefficients with the output of the credit allocation module, realizing collaborative communication for credit perception.

[0087] For Algorithm 3, the two-layer adaptive intrinsic reward algorithm, compared with existing methods such as stochastic network distillation and gradient-based intrinsic reward optimization, this invention has the following distinguishing features: the stochastic network distillation method only serves exploration-driven processes and is not coupled with the collective goal; while the single-layer coupled gradient-based optimization method achieves learnable intrinsic rewards, it does not associate the reward symbol with multi-agent credit allocation; this invention anchors the intrinsic reward symbol to the credit gradient, enabling intrinsic rewards to not only encourage exploration but also exploration beneficial to the collective, thus unifying exploration incentives and credit allocation. Furthermore, this invention employs a two-layer nested scheduling of individual learning progress and collective performance, while existing methods only use single-layer decay or fixed weights.

[0088] The following examples fully describe the application environment, algorithm deployment method, parameter configuration and execution effect for highway overload management scenarios of different scales and technical focuses.

[0089] Example 1: Multi-department Collaborative Governance System for Provincial Expressway Network

[0090] This embodiment applies to the expressway network of a certain province, which has a total expressway mileage of over 5,000 kilometers and an average daily freight volume of 80,000 vehicles. Overweight vehicle management involves multiple departments, characterized by different hierarchical relationships, significant differences in objective functions, and complex communication links among these departments.

[0091] This embodiment employs a deployment architecture that combines centralized training with edge execution. During the training phase, the system is deployed on a cloud computing platform, equipped with four GPU accelerator cards to batch process real-time data transmitted from toll stations, inspection stations, and vehicle control points throughout the province. During the execution phase, edge inference nodes are deployed in each city-level branch center. Each city-level node only runs the policy network of the intelligent agents within its jurisdiction, without relying on the central node for real-time communication.

[0092] The observation space configuration for each department's intelligent agents is as follows:

[0093] The intelligent agent of the comprehensive management department observes the number of overweight vehicles, the distribution of vehicle types, the historical detection rate, and the real-time location and duty status of law enforcement personnel at each detection station.

[0094] Traffic management department's intelligent agent: The monitoring content includes real-time traffic speed, accident rate, traffic control information, and saturation of key road sections in the jurisdiction's road network;

[0095] The maintenance department's intelligent agent observes pavement damage index, bridge health monitoring data, historical maintenance records, and current maintenance resource reserves.

[0096] Operations department intelligent agent: The observation content includes toll revenue, toll station queue length, ETC transaction success rate, and holiday traffic forecast;

[0097] For the large-scale scenario in this embodiment, the parameter configurations of the three algorithms are as follows:

[0098] In the BACA module of Algorithm 1, the number of departments N is set to 4, the learning rate for updating credit weights is set to 0.001, and the threshold for the credit gating function is set to -0.1—when a department's... A value less than -0.1 is considered a negative contribution, activating the negative credit flow branch. Both the positive and negative credit networks are three-layer fully connected networks, with hidden layer dimensions set to 256 and 128 respectively, to adapt to the complex nonlinear characteristics of the value function in provincial governance. The credit sparsity regularization coefficient is set to 0.01 to encourage the concentration of credit weights in core decision-making departments.

[0099] In the GAC module of Algorithm 2, the number of multi-head attention heads M is set to 8, and the dimension of each attention head is 32. The graph attention network has 3 stacked layers, so that the receptive field of each department agent can cover all four provincial departments and the major municipal departments within its jurisdiction. The rules for constructing the neighbor node set are as follows: when the attention coefficient is greater than the threshold of 0.2, department j is included in the neighbor set of department i; if the sum of the attention coefficients of a department is continuously lower than 0.3 for more than 5 time steps, the island mode is triggered. The communication efficiency regularization coefficient is set to 0.05, and the attention threshold is set to 0.15.

[0100] In the BAIR module of Algorithm 3, the intrinsic reward generation network adopts a three-layer structure with a hidden layer dimension of 128 and an output dimension of 1. The value change threshold is set to 0.05, meaning that a single step change in individual value exceeding 5% is counted as a valid learning step. The collective performance decay rate is set to 0.02, ensuring that the intrinsic reward weight remains at a high level in the early stages of training to encourage exploration, and gradually decays after performance stabilizes. The collective performance index is defined as the moving average of the weighted composite score over the past 100 rounds, with the weight of the number of overweight vehicles detected being 0.4, the decrease in road damage index being 0.3, the traffic efficiency maintenance coefficient being 0.2, and the departmental collaboration satisfaction index being 0.1.

[0101] The training phase employed intensive training, lasting 150,000 rounds. Each round simulated 48 consecutive hours of highway traffic operation, with a time granularity of 15 minutes. The first 50,000 rounds utilized the full BAIR intrinsic reward mechanism. The initial value is set to 1.0, and the overall performance improvement rate ΔJ is increased. t or The loss gradually decreases as the number of rounds increases; in the middle 50,000 rounds, λ2 is linearly decreased from 0.15 to 0.05, allowing the model to gradually transition from relying on auxiliary loss guidance to autonomous optimization; in the last 50,000 rounds, only the temporal difference main loss and credit allocation loss are retained for fine-tuning.

[0102] After training convergence, the policy networks of the four provincial departments and their respective municipal branches are deployed to edge inference nodes. During the execution phase, only the forward propagation of the policy networks is run, with inference latency controlled within 30 milliseconds. Retraining is triggered when the following conditions occur: the collective performance index drops by more than 15% from the best value for seven consecutive days, or the credit weight of a department remains at 0 for more than 48 hours.

[0103] The following technical effects were achieved after this embodiment was actually deployed in a certain province:

[0104] The detection rate of overweight vehicles increased from 61% before the deployment to 89%;

[0105] The response time for joint decision-making by multiple departments has been reduced from an average of 35 minutes to 8 seconds.

[0106] Road maintenance costs due to overloading decreased by approximately 26% annually;

[0107] The efficiency of truck traffic at toll stations has increased by 18%, and the average queue length has decreased by approximately 40%.

[0108] The interdepartmental collaboration satisfaction index increased from 0.52 to 0.87.

[0109] Example 2: Overweight Management System for Typical Urban Transit Corridors.

[0110] This embodiment applies to an outer ring road of an industrial city. The road is 85 kilometers long, connecting three logistics parks and two port terminals, with an average daily truck traffic volume of 23,000 vehicles. The overweight rate has consistently remained between 18% and 25%. Compared to Embodiment 1, this scenario involves fewer departments, but their interests are incompatible—relevant departments, due to limited manpower, tend to reduce the frequency of inspections to ensure smooth traffic on the main road, directly conflicting with the goal of controlling overweight vehicles. This provides a typical verification scenario for the non-monotonic credit allocation of this invention.

[0111] This embodiment employs a fully edge-based deployment architecture, independent of a central cloud node. One inference server is deployed in each of the four department offices, interconnected via the external network. Departments only exchange embedding vectors filtered by the graph attention mechanism, avoiding the sharing of raw observation data to ensure data security. The training phase combines offline training with online fine-tuning: pre-training is first completed on the provincial platform, then the pre-trained model parameters are distributed to each city-level edge node, and incremental fine-tuning is performed every 24 hours using local data.

[0112] The observation space configuration for each department's intelligent agents is as follows:

[0113] The intelligent agent of the comprehensive management department observes real-time overweight data from 3 fixed detection stations and 2 mobile detection points, law enforcement personnel's duty roster, and a blacklist of overweight vehicles.

[0114] Traffic police department intelligent agent: The monitoring content includes real-time vehicle speed, congestion index, accident alarms, and police deployment status at each section of the outer ring road;

[0115] The maintenance department's intelligent agent observes the pavement condition index (PCI), bridge and culvert health scores, and the progress of daily maintenance plans.

[0116] Operations Department Intelligent Agent: The monitoring content includes toll revenue, ETC lane utilization, holiday traffic forecast, and toll collection staff scheduling.

[0117] In view of the characteristics of this embodiment, such as a small number of departments and significant conflicts of interest, the algorithm parameters are adjusted as follows:

[0118] The BACA module in Algorithm 1 is the core innovation, focusing on resolving the conflict between relevant departments' goals of reducing interception rates and controlling overloading. The hidden layer dimensions of the positive and negative credit networks are reduced to 64 and 32, respectively. The credit gating threshold is set to -0.05 to lower the trigger threshold for negative contribution judgment, enabling more timely identification and suppression of negative behaviors by relevant departments. The negative suppression weight update learning rate η is set to 0.005, five times that in Example 1, to accelerate the learning response to negative behaviors. The credit sparsity regularization coefficient is set to 0.02 to ensure that the credit weights can significantly distinguish between positive and negative departments.

[0119] In the GAC module of Algorithm 2, the number of multi-head attention heads M is set to 4. Due to the small number of departments, the number of stacked layers in the graph attention network is reduced to 2, and the communication efficiency regularization coefficient is set to 0.02. The trigger threshold for the island mode is set to the sum of attention coefficients being below 0.5 for more than 3 time steps. To adapt to the bandwidth limitations of the municipal government extranet, the dimension of the embedded vector is compressed to 32 dimensions, and the amount of data transmitted between departments is reduced to one-quarter of that in Example 1.

[0120] In the BAIR module of Algorithm 3, the value change threshold is set to 0.03, and the collective performance decay rate is set to 0.05—a higher decay rate allows intrinsic rewards to give way to extrinsic rewards more quickly, which aligns with the short training cycle characteristic of this scenario. The collective performance index J_t is defined as the weighted harmonic mean of the number of overweight vehicle checks and traffic efficiency, aiming to balance the conflicting demands of governance goals and traffic flow assurance. A traffic efficiency maintenance coefficient is added to the reward function, with a weight set to 0.3, to guide traffic police departments to find the optimal balance between ensuring smooth traffic flow and implementing overweight vehicle checks.

[0121] The training process is divided into three stages, with a total of 50,000 training rounds.

[0122] Phase 1: Pre-training the model on a provincial platform using historical data for 20,000 rounds. In this phase, the initial exploration rate of the traffic police department's agent is set to 0.5, higher than other departments, allowing it to fully explore the two drastically different strategy spaces of increasing and decreasing inspection rates, accumulating experience samples in conflict scenarios. The initial intrinsic reward coefficient is set to 2.0, providing stronger exploration incentives to guide the model to find the Pareto optimal boundary.

[0123] The second phase involves distributing the pre-trained model to city-level edge nodes and fine-tuning it using local data for 10,000 rounds. During this phase, the credit gating threshold is gradually adjusted from -0.05 to -0.08, making the suppression of negative credit flows by the traffic police department more stringent. Simultaneously, λ² is gradually decayed from 0.1 to 0.05, reducing the reliance on graph attention communication and allowing the policy network to gradually adapt to a distributed execution mode of independent decision-making.

[0124] Phase 3: Continuous fine-tuning after online deployment. Every 24 hours, five rounds of incremental updates are performed on the policy network using the day's generated detection data, with no more than 200 update steps to avoid catastrophic policy obsolescence. During incremental updates, the parameters of the BAIR module are frozen, and only the BACA and GAC modules are updated, as the intrinsic rewards have stabilized during the deployment phase and require no further adjustment.

[0125] The execution phase employs a deterministic strategy, directly taking the maximum probability output of the policy network as the decision action, without introducing exploratory noise. The inference latency for each agent is controlled within 15 milliseconds, and the inter-department embedding vector exchange cycle is 10 seconds.

[0126] Offline retraining is triggered when the following situations occur: the rate of overweight vehicle detection is less than 65% for 3 consecutive days, or the negative credit weight of the traffic police department exceeds 0.4 for 7 consecutive days, indicating that too much negative behavior has accumulated and the credit allocation has significantly deviated from the equilibrium point.

[0127] The following technical effects were achieved after this embodiment was deployed in the city's transit corridor for 6 months:

[0128] The detection rate of overweight vehicles increased from 51% to 76%;

[0129] Traffic police increased the frequency of proactive checks from an average of 42 times per day to an average of 68 times per day, resulting in a reduction of negative behaviors by approximately 60%.

[0130] The average traffic speed on the outer ring road decreased by only 3.2%, achieving a balance between governance goals and traffic efficiency;

[0131] The frequency of road maintenance due to overloading decreased by 42%, saving the maintenance department approximately 1.8 million yuan in annual budget.

[0132] Convergence speed: Compared to the control group that does not use BAIR intrinsic rewards, the number of rounds required for policy convergence is reduced from 42,000 to 18,000, an acceleration of approximately 57%.

[0133] Example 3: Overweight instant interception system at highway toll station entrances.

[0134] This embodiment is applied to a single large toll station on a highway, which handles an average of 5,000 trucks per day and has four entrance weighing lanes. Overweight vehicle management involves four roles: toll station personnel responsible for controlling overweight vehicles, traffic police officers stationed on the road section, maintenance and inspection personnel, and toll station supervisors. The difference between this embodiment and embodiments one and two is that the intelligent agent is a hybrid role of human and equipment rather than a purely institutional department. The requirements for inference latency are more stringent, and the real-time requirements are the highest—tracking overweight vehicles after they leave the toll station is extremely difficult, requiring a decision to intercept or release the vehicle within 3 seconds of it passing the dynamic scale.

[0135] This embodiment employs an embedded deployment architecture on the device side, running the inference program directly on the industrial control computer at the toll station, without relying on network communication. The four roles access the system via mobile terminals or fixed workstations, and the agents exchange embedding vectors through a local area network, with communication latency required to be less than 10 milliseconds. Offline training is used during the training phase; after model training on historical data, the policy network is burned into the embedded device, eliminating the need for online learning. This embodiment does not deploy a centralized training environment; all training is completed in a laboratory setting, with only forward inference performed on-site.

[0136] The observation space configuration for each agent is as follows:

[0137] Intelligent agent for controlling overloaded vehicles: The observation content includes dynamic weighing data, vehicle axle type recognition results, historical overload records, and current lane queue length;

[0138] Traffic Police Intelligent Agent: The observation content includes real-time video analysis results of toll plazas, such as traffic density, violations, police on-duty status, and accident information in nearby road sections;

[0139] Maintenance inspector AI agent: The observation content includes the status of weighing equipment, road surface condition, and recent maintenance records;

[0140] The toll collection supervisor intelligent agent observes the number of lanes open, the status of toll collectors, information on special vehicle passes, and real-time toll revenue data.

[0141] To address the low latency and high real-time performance characteristics of this embodiment, the algorithm parameters have been simplified and optimized:

[0142] In the BACA module of Algorithm 1, the number of departments N is set to 4. To meet the real-time requirement of completing decisions within 3 seconds, the positive and negative credit networks are significantly simplified to single-hidden-layer networks with 32 hidden layer neurons. The activation function is ReLU instead of LeakyReLU to reduce computational overhead. Credit weight updates are not performed during the deployment phase—because online training is not performed in the field environment. and The credit gradient is frozen at the optimal value during training convergence and is not used in online inference. The credit gating threshold is set to -0.12, employing a wider negative decision range to avoid overly sensitive gating causing frequent policy switching in real-time environments. The credit gradient is only used to determine the reward sign during training and is not calculated during inference.

[0143] In the GAC module of Algorithm 2, the number of multi-head attention points M is set to 2, and only the attention coefficient related to the vehicle control personnel is calculated—focusing on capturing the interaction between vehicles in adjacent lanes, while attention calculations between other departments are pruned. The graph attention network is stacked with only one layer because the collaboration between departments in the toll station scenario is mainly based on direct interaction, without the need for multi-hop information transmission. To meet the 10-millisecond communication latency limit, the neighbor node set is predefined as a fixed topology rather than dynamically constructed, and all departments are fully connected pairwise. The communication efficiency regularization coefficient is set to 0. Attention is not sparsified in this scenario because the number of departments is small and communication is completed within the same local area network.

[0144] In the BAIR module of Algorithm 3, the value change threshold is set to 0.02. The collective performance decay rate is set to 0.08, because real-time scenarios require rapid policy development, and the intrinsic reward should anneal quickly in the early stages of training. The collective performance metric is defined as the product of the detection accuracy and interception success rate over the past 200 interception windows. The extrinsic reward function is significantly simplified—it only includes a positive reward for successfully detecting overweight vehicles and a negative reward for mistakenly intercepting normal vehicles, without any combined weights, because the multi-dimensional combination of reward signals in real-time scenarios increases the variance of policy evaluation and affects the convergence speed.

[0145] Training was conducted offline in a laboratory environment, with a total of 30,000 training rounds. Each round simulated a tollbooth operating for 4 consecutive hours, with a time granularity of 10 seconds. During training, time periods with a high frequency of overload events were selectively sampled as key training samples—time periods with an overload rate exceeding 30% were oversampled from the original data, allowing the model to fully learn the optimal strategy in scenarios with fewer samples within a long-tailed distribution.

[0146] For the first 10,000 rounds, the full BAIR intrinsic reward mechanism was used for warm-up training, with the initial intrinsic reward coefficient set to 1.5. For the next 15,000 rounds, λ² was linearly decayed from 0.05 to 0.01, gradually reducing the graph attention communication intensity, as the policy tends to stabilize in the later stages of training, and the need for inter-departmental information exchange decreases accordingly. For the final 5,000 rounds, the BAIR module parameters were frozen, with only the BACA and GAC modules fine-tuned.

[0147] After training convergence, model pruning and quantization operations are performed: network connections with attention coefficients less than 0.1 are removed, network parameters are quantized from 32-bit floating-point to 8-bit integers, and the model size is compressed from 12MB to 1.5MB. The quantized policy network is then burned into the embedded AI chip of the toll station's industrial control computer, with single inference latency controlled within 8 milliseconds.

[0148] The following technical effects were achieved after this embodiment was deployed at the toll station:

[0149] The success rate of intercepting overweight vehicles increased from 71% to 94%;

[0150] The average end-to-end delay from when a vehicle passes the dynamic scale to when an intercept or release command is issued is 2.3 seconds, which meets the design requirement of within 3 seconds.

[0151] The rate of false toll booths was reduced from 3.5% to 0.8%, thus reducing congestion and disputes in toll lanes caused by false toll booths.

[0152] Embedded devices have a CPU utilization rate of less than 15% and a memory usage of less than 4MB, and can coexist with other industrial control tasks without interfering with each other.

[0153] It has no network dependency and can still work normally in extreme situations such as interruption of the government extranet, with system availability reaching 99.95%.

[0154] The above technical solutions only embody the preferred technical solutions of the present invention. Any modifications that may be made by those skilled in the art to certain parts thereof embody the principles of the present invention and fall within the protection scope of the present invention.

Claims

1. A collaborative decision-making and handling method for highway overload management based on multi-agent reinforcement learning, characterized in that, Includes the following steps: The transportation management departments, traffic management departments, road maintenance departments, and operation management departments involved in the governance of overweight vehicles on highways are modeled as heterogeneous intelligent agents; A multi-agent reinforcement learning framework with centralized training and distributed execution is constructed. During the training phase, global state information is used for joint policy optimization, while during the execution phase, each agent makes independent decisions based only on local observations. Through the bidirectional attention credit allocation module, the positive contribution weight and negative inhibition weight of each department's decision to the joint goal are dynamically calculated. The individual action value of the positive contribution department is aggregated through the positive credit flow, and the individual action value of the negative contribution department is inhibited through the negative credit flow. Finally, the joint action value is generated through nonlinear mapping. The graph attention collaborative decision-making module dynamically constructs a departmental relationship graph based on the hidden state of each department, calculates the attention coefficient between departments, and enables each department to only aggregate information from neighboring departments with high correlation for policy updates, thereby achieving efficient collaboration in a partially observable environment. Through a two-layer adaptive intrinsic reward module, continuous intrinsic reward signals are generated for each department. The inner layer tracks the learning progress of each department, while the outer layer dynamically adjusts the weight coefficient of the intrinsic reward based on the collective performance improvement rate, thus solving the problem of strategy learning difficulties under sparse external rewards. Based on the converged policy network, the system receives observation data from various departments in real time and outputs coordinated actions from these departments. These actions include instructions to intercept overweight vehicles, instructions to inspect source enterprises, instructions to dispatch road maintenance, and instructions to collect tolls.

2. The method according to claim 1, characterized in that, The bidirectional attention credit allocation module adopts a credit flow separation mechanism. When the individual decision contribution of a department is lower than a preset threshold, the negative credit flow branch is activated to deduct the contribution of that department from the joint value. The non-continuous switching between the positive credit flow and the negative credit flow is achieved through a gating function. The negative inhibition weights are updated based on the credit gradient of each sector, which measures the marginal contribution of individual value to the joint value, so that the degree of negative inhibition can adaptively respond to the actual direction of the impact of individual behavior on the collective goal.

3. The method according to claim 1, characterized in that, In the graph attention collaborative decision-making module, the topology of the inter-department relationship graph evolves dynamically over time. The attention coefficient is calculated using a multi-head attention mechanism, where each attention head independently calculates the correlation weight between departments. The outputs of each attention head are spliced ​​or averaged to obtain the final aggregation result. When the total attention coefficient of a department falls below a preset threshold, the department enters an island mode, suspends receiving information from neighboring departments, and the agent no longer aggregates any neighboring information. Its policy input is only the local hidden state, and the graph attention aggregation step is skipped to achieve system fault tolerance when communication quality deteriorates.

4. The method according to claim 1, characterized in that, In the dual-layer adaptive intrinsic reward module, the intrinsic reward function is generated by a learnable neural network. The optimization objective of this network is to maximize the positive correlation with the collective value function, and the sign direction of the intrinsic reward is determined by the sign of the partial derivative of the collective value with respect to the individual value. This allows the intrinsic reward to adaptively switch between incentive and penalty modes based on the individual's marginal contribution to the collective. The adaptive coefficients are updated in the inner layer based on the temporal change amplitude of the individual value function, and updated in the outer layer using exponential decay scheduling. The decay rate is positively correlated with the normalized improvement rate of the collective performance, thereby achieving dual-layer coupled scheduling of individual learning progress and collective performance.

5. The method according to claim 1, characterized in that, The algorithm modules are coupled with gradients through a unified joint loss function. The loss of the credit allocation module depends on the enhanced features output by the graph attention collaborative decision-making module, the policy gradient of the graph attention collaborative decision-making module depends on the total reward generated by the two-layer adaptive intrinsic reward module, and the reward sign of the two-layer adaptive intrinsic reward module depends on the credit gradient output by the credit allocation module, forming a joint optimization mechanism that is interlocked among the three. In the centralized training phase, a weighted multi-loss function is used for joint optimization. The credit allocation loss, collaborative decision-making loss, and intrinsic reward generation loss are superimposed with the main temporal difference loss according to their respective weight coefficients. The weight coefficients of each loss term are adaptively adjusted during training according to the convergence progress of each module.

6. The method according to claim 1, characterized in that, The distributed execution phase employs an ε-greedy strategy for action selection, where the exploration rate ε decays exponentially with the number of training rounds. Each agent relies only on local observations and the embedding information of neighboring departments received through the graph attention mechanism during execution, without relying on the global state. After training convergence, only the policy network of each agent is retained during actual deployment, eliminating the need for graph attention aggregation calculations and intrinsic reward generation calculations, and controlling the single inference latency to within fifty milliseconds.

7. The method according to claim 1, characterized in that, The action space of each heterogeneous agent includes a mixture of continuous parameters and discrete instructions. The continuous parameters include detection thresholds and dispatch quantity, while the discrete instructions include interception instructions, release instructions, inspection instructions, and scheduling commands. The mixture is decoupled into continuous branches and discrete branches, which are then output separately and then merged for execution.

8. The method according to claim 1, characterized in that, The external reward function includes at least a weighted combination of the following dimensions: the number of overweight vehicles detected, the decrease in road damage index, the traffic efficiency maintenance coefficient, and the departmental collaboration satisfaction index, wherein the departmental collaboration satisfaction index is calculated by the similarity between the actual actions performed by each department and the joint optimal action.

9. The method according to claim 1, characterized in that, During the observation encoding process in the training phase, each agent fuses the current local observation with the historical hidden state through a gated recurrent unit encoder to generate the current hidden state representation. In the graph attention-driven information aggregation process, each agent calculates the attention coefficient with the current hidden state as the node feature, and generates the enhanced hidden state by aggregating highly correlated information from neighboring departments. During the credit allocation process, the enhanced hidden state, executed actions, total rewards, and historical states of each department are stored in the experience replay pool. After sampling batch data from the replay pool, the credit weight and joint action value are calculated.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 9.

Images