Concrete dynamic scheduling method based on multistage reinforcement learning and intelligent agent simulation

Abstract

The invention relates to the field of concrete dynamic scheduling, in particular to a concrete dynamic scheduling method based on multistage reinforcement learning and intelligent agent simulation. According to the technical scheme, the method comprises the following steps: constructing an ABM microscopic traffic simulation environment for concrete dynamic scheduling; designing a layered reinforcement learning framework of concrete dynamic scheduling; a DDQN strategy network is trained; and triggering a concrete dynamic scheduling mechanism. According to the invention, environment change is responded in real time through a dynamic decision-making mechanism of hierarchical reinforcement learning. And the microscopic traffic behavior modeling precision is improved. According to the method, the two-way single-lane scene is finely simulated through ABM, random deceleration, overtaking decision making and the like are included, the transportation time prediction deviation rate is reduced, the transportation cost estimation error is reduced through microscopic behavior modeling, and the fuel consumption optimization rate is increased. According to the method, a double-layer full-connection feature extractor is introduced, so that nonlinear features in a production-transportation coupling relation are effectively captured. The method is suitable for concrete dynamic scheduling.

Description

Technical Field

[0001] The present invention relates to the field of concrete dynamic scheduling, and specifically relates to a concrete dynamic scheduling method based on multi-level reinforcement learning and agent simulation. Background Art

[0002] In large-scale infrastructure projects such as high-speed railway construction, the dynamic scheduling of concrete is a core issue affecting project efficiency and cost. Traditional scheduling methods mostly adopt static optimization strategies (such as genetic algorithms, mixed integer programming) or simple heuristic rules, and it is difficult to cope with dynamic demand fluctuations, traffic environment uncertainties, and the complexity of microscopic vehicle behavior.

[0003] In the prior art, for example, "Cooperative Scheduling of Distribution Vehicles in Concrete Mixing Plants" proposed a cooperative scheduling scheme for multiple mixing plants based on an improved genetic algorithm. By optimizing the vehicle distribution sequence and time in the form of double-layer coding, the interruption pouring and waiting time at the construction site are reduced. This research modeled the problem as a vehicle routing problem with time windows, aiming to minimize the total waiting time, used a real-number coding genetic algorithm to generate a scheduling scheme, and verified the scheduling effect through Gantt charts. This method achieved good optimization results in a static environment by coordinating the resource allocation of multiple mixing plants.

[0004] However, the existing methods have the following limitations:

[0005] (1) Insufficient dynamic adaptability: The model assumes that parameters such as transportation time and road conditions are fixed, and it cannot respond to sudden demand changes (such as a sudden increase in demand at the construction site) or emergencies such as traffic congestion in real time, resulting in the failure of the scheduling scheme in a dynamic environment.

[0006] (2) Lack of microscopic behavior modeling: The microscopic behavior of vehicles in restricted roads (such as two-way single-lane roads) (random deceleration, overtaking decisions) is not considered, and only macro transportation time estimation is relied on, resulting in a deviation between the scheduling result and the actual execution (the experiment shows that the deviation rate of transportation time is as high as 25.6%).

[0007] Low computational efficiency: The genetic algorithm faces the problem of solution space explosion when dealing with large-scale problems, with a large number of iterations (more than 1000 times) and a slow convergence speed, making it difficult to meet the real-time scheduling requirements.

[0008] (3) Difficulty in multi-objective trade-off: Using the linear weighted method to combine multiple objectives (transportation cost, waiting time, demand satisfaction rate) into a single fitness function, it is impossible to effectively balance conflicting objectives in complex scenarios. For example, it is difficult to achieve an optimal trade-off between cost and construction period.

[0009] These defects stem from the insufficient representation ability of traditional optimization methods for high-dimensional dynamic state spaces and the lack of refined modeling of microscopic traffic interactions. Summary of the Invention

[0010] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a concrete dynamic scheduling method based on multi-level reinforcement learning and intelligent agent simulation, which realizes efficient scheduling of concrete in a dynamic environment.

[0011] The present invention adopts the following technical solutions to achieve the above-mentioned purpose. The present invention provides a concrete dynamic scheduling method based on multi-level reinforcement learning and intelligent agent simulation, comprising:

[0012] S1. Construct an ABM (Agent-Based Modeling) micro-traffic simulation environment for dynamic concrete scheduling;

[0013] S2. Design a hierarchical reinforcement learning framework for dynamic concrete scheduling;

[0014] S3. Train the DDQN (Double Deep Q-Network) policy network.

[0015] S4. Trigger the dynamic scheduling mechanism of concrete.

[0016] Furthermore, step S1 specifically includes:

[0017] Parameter initialization:

[0018] Input: Maximum speed limit V m , vehicle initial position, acceleration a, deceleration b, random deceleration probability p slow_down , meeting distance threshold distance_threshold;

[0019] Output: Initialize the micro-behavior rule base of the vehicle agent, including acceleration, deceleration, random deceleration, and lane avoidance logic;

[0020] Behavioral rules definition:

[0021] Acceleration rule: If the vehicle's current speed V i (t) <V m , then press V i (t+1)=V i (t)+a×t update speed;

[0022] Speed reduction rule: If the distance to the vehicle in front is d i-1 (t), then press V i (t+1)=min(V i (t),d i-1 (t)-1) deceleration;

[0023] Random deceleration: with probability p slow_down To trigger a deceleration event, press Vi (t + 1)=V i (t)-b×t update speed;

[0024] Meeting vehicle avoidance: When the distance between oncoming vehicles is less than distance_threshold, the lane change logic is triggered, and the vehicle enters the passing lane with a set probability and resumes the original lane after overtaking.

[0025] Furthermore, step S2 specifically includes:

[0026] High-level decision-making:

[0027] Input: Mixing plant production capacity C i , queue status QS i , construction site demand D j , received quantity RA j ;

[0028] Action space: Generate a mixing plant - construction site matching vector HLAV ∈ {0, 1,..., Ns × Nc - 1}, and the encoding rule is HLAV = station_idx × Nc + site_idx, where Ns represents the number of mixing plants, Nc represents the total number of construction sites, station_idx represents the index number of the mixing plant, and site_idx represents the index number of the construction site;

[0029] Low-level decision-making:

[0030] Input: Time window TW, discrete time interval T i , available time slots;

[0031] Action space: Generate a scheduling time vector LLAV ∈ {0, 1,..., TW / Ti - 1}, and the encoding rule is combined_action = HLAV × (TW / Ti)+LLAV.

[0032] Furthermore, step S3 specifically includes:

[0033] High-level DDQN: The input layer receives the mixing plant and construction site state features, the fully connected layer processes them, and the output layer outputs the Q-value matrix of the high-level DDQN;

[0034] Low-level DDQN: The input layer receives the time window features, the fully connected layer processes them, and the output layer outputs the Q-value matrix of the low-level DDQN;

[0035] The training process includes:

[0036] Experience replay: Store the state transition tuple (s t , a t , r t , s t+1) to the buffer, and batch sample to update the network weights;

[0037] Target network update: Synchronize the online network parameter θ to the target network θ' every set number of steps, and the update formula is θ′←τθ+(1-τ)θ′;

[0038] s t represents the current state, including the production capacity C of the mixing plant i , the vehicle queue status QS i , the demand D of the construction site j , the transportation time TT ij , the amount of concrete in transit ITA ij information, a t represents the current action, including high-level actions and low-level actions, r t represents the immediate reward, calculated according to the reward calculation formula, s t +1 represents the next state;

[0039] Reward calculation:

[0040] Composite reward R, R = αRcost + βRwait + γRdemand, where Rcost = -∑TC ij , Rwait = -∑WTC ij , Rdemand = ∑min(RA j , D j ).

[0041] TC ij represents the transportation cost, α represents the transportation cost weight, β represents the waiting time cost weight, and γ represents the demand satisfaction rate weight.

[0042] Furthermore, the dynamic adjustment method of the weights α, β, and γ is:

[0043] According to the urgency of the construction site demand and the road congestion status, adopt the adaptive weight update formula:

[0044]

[0045] Update β in the same way t+1 and γ t+1 , ensuring that the multi-objective optimization weights change dynamically with the environment.

[0046] Furthermore, step S4 specifically includes:

[0047] Event detection: Real-time monitor the change in construction site demand ΔD j or road abnormal events;

[0048] Rescheduling condition: If ∣ΔD j ∣ or D j> 15% or the average vehicle waiting time is greater than 30 minutes, triggering the policy network to re - make a decision;

[0049] Policy update: Generate a new scheduling plan based on the latest environmental state, and preferentially adjust the time slots and transportation routes of unexecuted tasks.

[0050] Furthermore, step S2 includes the design of a two - layer fully - connected feature extractor:

[0051] Adopt a two - layer fully - connected neural network. The input layer receives a high - dimensional state vector After being processed by two hidden layers with 256 neurons, it outputs a low - dimensional feature vector f∈R128f∈R128, which is used to capture the non - linear features in the production - transportation coupling relationship. The feature extractor is jointly trained with DDQN, and the weight matrices W1, W2 and bias vectors b1, b2 are optimized through the loss function.

[0052] Furthermore, the process of integrating ABM and DDQN in the step S4 concrete dynamic scheduling mechanism includes:

[0053] Set the initial vehicle position X i (0) and speed V i (0);

[0054] Every time the vehicle position X i (t) and speed V i (t) are updated, and the updated traffic density v(t), average vehicle speed The congestion section marker Fcongestion is used as the state input s t for DDQN to generate scheduling actions;

[0055] The vehicle queue state QS i is updated in real - time according to the number of tasks to be scheduled. The formula is:

[0056] QS i (t + 1)=QS i (t)-∑ j SA ij (t)+∑ k δ ik (t), where δ ik (t) is the new task indication function, and SA ij (t) represents the total amount of concrete dispatched from the mixing plant to the construction site within time t.

[0057] The beneficial effects of the present invention are as follows:

[0058] Through the dynamic decision - making mechanism of hierarchical reinforcement learning, the present invention can respond to environmental changes in real - time and improve the accuracy of microscopic traffic behavior modeling.

[0059] Through the refined simulation of two-way single-lane scenarios by ABM, including random deceleration, overtaking decisions, etc., the present invention reduces the prediction deviation rate of transportation time, and the microscopic behavior modeling reduces the estimation error of transportation costs and improves the optimization rate of fuel consumption.

[0060] The present invention introduces a double-layer fully connected feature extractor to effectively capture the non-linear features in the production-transportation coupling relationship. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flowchart of a concrete dynamic scheduling method based on multi-level reinforcement learning and agent simulation provided by the present invention;

[0062] Figure 2 is a specific algorithm flowchart of the dynamic scheduling provided by the present invention;

[0063] Figure 3 is a network topology structure diagram of the reinforcement learning provided by the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0065] The present invention provides a concrete dynamic scheduling method based on multi-level reinforcement learning and agent simulation, as Figure 1 、 Figure 2 shown, specifically including:

[0066] Step 1: Construct an ABM microscopic traffic simulation environment;

[0067] (1) Parameter initialization:

[0068] Input: Maximum road speed limit V m 、Vehicle initial position X i (0)、Acceleration a, deceleration b, random deceleration probability p slow_down 、Meeting distance threshold distance_threshold.

[0069] Output: Initialize the microscopic behavior rule library of vehicle agents (including acceleration, deceleration, random deceleration, and meeting lane avoidance logic).

[0070] (2) Definition of behavior rules:

[0071] Acceleration rule: If the current speed of the vehicle V i (t) < V m , then update the speed according to V i (t + 1) = V i (t) + a * t.

[0072] Deceleration rule: If the distance d i-1 from the vehicle in front at time t i , then decelerate according to V i (t) such that V i-1 (t + 1) = min(V

[0073] Random deceleration: With probability p slow_down , trigger a deceleration event and update the speed according to V i (t + 1) = V i (t) - b * t.

[0074] Meeting and giving way: When the distance between oncoming vehicles is less than distance_threshold, trigger the lane-changing logic. The vehicle enters the passing lane with probability lane_change_probability, and resumes the original lane after overtaking.

[0075] Step 2: Design of the hierarchical reinforcement learning framework;

[0076] The hierarchical reinforcement learning framework of the present invention is as Figure 3 shown:

[0077] High-level decision-making (HLAV):

[0078] Input: The production capacity C of the mixing plant Mi i , the queue status QS i , the demand D j of the construction site S j , and the received quantity RA j .

[0079] Action space: Generate a mixing plant - construction site matching vector HLAV ∈ {0, 1,..., Ns × Nc - 1}, and the encoding rule is HLAV = station_idx × Nc + site_idx, where Ns represents the number of mixing plants, Nc represents the total number of construction sites, station_idx represents the index number of the mixing plant, and site_idx represents the index number of the construction site;

[0080] Low-level decision-making (LLAV):

[0081] Input: Time window TW, discrete time interval T i , and the available time slot ATSk.

[0082] Action space: Generate a scheduling time vector LLAV ∈ {0, 1,..., TW / Ti - 1}, and the encoding rule is combined_action = HLAV × (TW / Ti) + LLAV.

[0083] Step 3: Training of the DDQN policy network

[0084] Network structure:

[0085] High-level DDQN: Input layer (status features of mixing plant and construction site) → Fully connected layer (256 neurons, ReLU) → Output layer (Q-value matrix QHigh-Level).

[0086] Low-level DDQN: Input layer (time window features) → Fully connected layer (256 neurons, ReLU) → Output layer (Q-value matrix Q Low-Level )

[0087] Training process:

[0088] Experience replay: Store the state transition tuple (s t , a t , r t , s t +1) into the buffer D, and update the network weights by batch sampling.

[0089] Target network update: Synchronize the online network parameters θ to the target network θ' every 1000 steps, and the update formula is θ′←τθ+(1 - τ)θ′ (τ = 0.005).

[0090] s t represents the current state, including the production capacity C of the mixing plant Mi i , the vehicle queue status QS i , the demand D of the construction site S j , the transportation time TT j , the in-transit concrete volume ITA ij information, a ij represents the current action, including the high-level action: selecting a mixing plant-construction site pairing, and the low-level action: selecting a specific scheduling time, r t represents the immediate reward, calculated according to the reward calculation formula, s t +1 represents the next state, the new state after executing the action, for example: the vehicle position update X t (t + 1), the received concrete volume RA at the construction site i increases, the change in the road congestion status, etc. j

[0091] Reward calculation, composite reward:

[0092] R = αRcost + βRwait + γRdemand;

[0093] Rcost = -∑TC ij , Rwait = -∑WTC ij , Rdemand = ∑min(RA j , D j )​

[0094] TC ij represents the transportation cost from the mixing plant Mi to the construction site S j The single - trip transportation cost, such as fuel cost, labor cost, etc., and the calculation formula is: Rcost=-∑TC ij .

[0095] α represents the weight of transportation cost, which balances the importance of transportation cost in the total reward. For example, if α = 0.5, it means that the transportation cost accounts for 50% of the total reward.

[0096] β represents the weight of waiting - time cost, which balances the importance of vehicle waiting - time cost. For example, if β = 0.3, it means that the waiting - time cost accounts for 30%.

[0097] γ represents the weight of demand - satisfaction rate, which balances the importance of the completion rate of construction - site demand. For example, if γ = 0.2, it means that the demand - satisfaction rate accounts for 20%.

[0098] By adjusting the weight coefficients α, β, γ, the model can flexibly balance in multi - objective optimization:

[0099] Minimize cost (dominated by α): Prioritize the path with low transportation cost;

[0100] Reduce waiting time (dominated by β): Avoid vehicle queuing at the construction site;

[0101] Maximize demand - completion rate (dominated by γ): Ensure sufficient concrete supply at the construction site.

[0102] Step 4: Trigger the dynamic scheduling mechanism

[0103] Event detection: Real - time monitor the change in construction - site demand ΔD j or road - anomaly events (such as congestion probability pcongestion>0.7).

[0104] Rescheduling condition: If ∣ΔD j ∣ / D j >15% or the average vehicle waiting time avg_wait_time>30 minutes, trigger the policy network to re - make a decision.

[0105] Policy update: Generate a new scheduling plan based on the latest environmental state, and preferentially adjust the time slots and transportation paths of unexecuted tasks.

[0106] Step 5: Optimize the feature extractor

[0107] Feature dimensionality reduction: Input the high - dimensional state vector x∈Rn→fully - connected layer (256 neurons, ReLU)→feature vector f∈R128.

[0108] Nonlinear mapping: The key features are extracted through a two-layer neural network \(f = \sigma(W_2\sigma(W_1x + b_1)+b_2)\) to enhance the generalization ability of DDQN for complex state spaces.

[0109] Technical solution verification conditions:

[0110] Hardware: NVIDIA RTX 3080 GPU, 32GB of memory, Intel i7-10700K CPU.

[0111] Software: Python 3.8, PyTorch 1.9.0, The ABM framework is implemented based on the Mesa library.

[0112] Parameter settings: Learning rate \(\alpha = 0.001\), discount factor \(\gamma = 0.99\), exploration rate \(\epsilon\) linearly decays from 1 to 0.1.

[0113] The present invention will be further described below in conjunction with specific implementation cases.

[0114] (1) HRL (Hierarchical Reinforcement Learning, hierarchical reinforcement learning framework)

[0115] Static relationship: The dynamic scheduling problem is decomposed into a macro layer (resource allocation) and a micro layer (vehicle scheduling), which are processed by high / low-level DDQN networks respectively. The macro layer is responsible for matching the mixing plant with the construction site, and the micro layer optimizes the specific departure time.

[0116] Dynamic relationship: Through ABM, the real-time behavior of vehicles on a two-way single-lane (acceleration, deceleration, passing and overtaking) is simulated, the environmental state (vehicle position, speed, in-transit concrete volume) is dynamically updated, and fed back to the reinforcement learning agent.

[0117] Effect:

[0118] Feasibility: The hierarchical structure reduces the decision-making dimension and avoids a single strategy falling into a local optimum.

[0119] (2) Agent-based modeling (ABM)

[0120] Key parameters: Maximum vehicle speed (\(V_m = 40\) km / h), random deceleration probability (\(p_{slow\_down}=0.2\)), overtaking speed increment (5 km / h). The vehicle spacing threshold (\(distance\_threshold = 50\) m) triggers the passing lane decision.

[0121] Dynamic simulation: The microscopic traffic flow is simulated through the acceleration and deceleration formula to generate high-fidelity state feedback.

[0122] Effect:

[0123] Feasibility: ABM accurately captures the impact of single-lane congestion on transportation time and supports the optimization of reinforcement learning strategies.

[0124] Beneficial effects: The deviation rate of transportation time simulated by ABM is only 5.5%, which is better than that of traditional integer models.

[0125] (3) Feature extractor

[0126] Structural design: A two-layer fully connected neural network (input layer → 256 ReLU → 256 ReLU → output layer) processes high-dimensional state data (such as the production capacity of the mixing plant and the amount of concrete in transit).

[0127] Dynamic optimization: Jointly trained with DDQN, the feature weights are optimized through the loss function.

[0128] Function and effect:

[0129] Feasibility: The feature extractor compresses the input dimension from n to m (m < n), accelerating the convergence of the model.

[0130] Beneficial effects: After adding the feature extractor, the loss value of the model approaches 0.01 after 500 rounds of training, and the convergence speed is increased by 37.5%.

[0131] (4) Dynamic scheduling mechanism

[0132] Trigger conditions: Sudden changes in construction site demand (such as a 20% increase in Dj), temporary road closures (injected through ABM events).

[0133] Response strategy: The RL agent reallocates the production capacity of the mixing plant and adjusts the departure time window based on the real-time state (RAj, ITAij).

[0134] Function and effect:

[0135] Feasibility: The dynamic adjustment strategy increases the demand satisfaction rate from 78% to 95%.

[0136] Beneficial effects: In the experimental case, when the demand at Site S7 suddenly increases from 267 to 320 m 3 the delay time is reduced by 43%.

[0137] (2) Effect experiment and data analysis

[0138] (1) Experimental settings

[0139] Hardware environment: Intel i7-10700K CPU + NVIDIA RTX 3080 GPU, Python 3.8 + PyTorch 1.9.

[0140] Data source: A high-speed rail section in southwestern China, 5 mixing plants (M1 - M5) and 8 construction sites (S1 - S8). Specific parameters are shown in the following table.

[0141] Table of equipment configuration parameters

[0142] Parameter M1 M2 M3 M4 M5 <![CDATA[Production capacity (m 3 )]]> 10 10 12 8 12 Number of transport vehicles 16 18 16 13 20 Working hours 4 4 4 4 4 Operating cost 100 100 100 100 100

[0143] Table of construction site parameters

[0144]

[0145] 2. Experimental results

[0146] ① Optimization of transportation cost: Comparison with benchmarks (traditional integer programming model, static genetic algorithm (GA))

[0147] Method Total cost (yuan) Cost reduction rate Traditional integer model 9765 - Static GA 8320 14.8％ This solution (ABM + HRL) 7560 22.5％

[0148] ② Improvement of time accuracy:

[0149] The deviation rate of transportation time in ABM simulation is only 5.5%, while that of the traditional model reaches 25.6%.

[0150] ③ Dynamic demand response:

[0151] When the demand at S7 suddenly increases, this solution shortens the delay from 4.2 hours to 2.4 hours through dynamic scheduling.

[0152] ④ Model convergence efficiency:

[0153] After adding the feature extractor, the number of training rounds is reduced by 37.5%, and the fluctuation of the loss value is reduced by 60%.

[0154] The above are only the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein, should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be changed within the scope of the concept described herein through the above teachings or the technology or knowledge in related fields. And any changes and modifications made by those skilled in the art without departing from the spirit and scope of the present invention shall fall within the protection scope of the appended claims of the present invention.

Claims

1. A concrete dynamic scheduling method based on multi-level reinforcement learning and agent simulation, characterized in that, Including: S1. Construct an ABM microscopic traffic simulation environment for concrete dynamic scheduling; S2. Design a hierarchical reinforcement learning framework for concrete dynamic scheduling; S3. Train a DDQN policy network; S4. Trigger the concrete dynamic scheduling mechanism.

2. The concrete dynamic scheduling method based on multi-level reinforcement learning and agent simulation according to claim 1, characterized in that Step S1 specifically includes: Parameter initialization: Input: Maximum speed limit V of the road m , Initial position of the vehicle, acceleration a, deceleration b, random deceleration probability p slow_down , Meeting distance threshold distance_threshold; Output: Initialize the microscopic behavior rule library of vehicle agents, and the rule library includes acceleration, deceleration, random deceleration, and passing lane avoidance logic; Behavior rule definition: Acceleration rule: If the current speed V of the vehicle i (t) < V m , then update the speed according to V i (t + 1) = V i (t) + a × t; Deceleration rule: If the distance d i-1 (t) from the vehicle ahead, then decelerate according to V i (t + 1) = min(V i (t), d i-1 (t) - 1); Random deceleration: with probability p slow_down trigger a deceleration event, and press V i (t + 1)=V i (t)-b×t to update the speed; Passing lane avoidance: When the distance between oncoming vehicles is less than the distance_threshold, trigger the lane change logic, and the vehicle enters the passing lane with a set probability and resumes the original lane after overtaking.

3. The concrete dynamic scheduling method based on multi-level reinforcement learning and agent simulation according to claim 1, wherein Step S2 specifically includes: High-level decision-making: Input: Mixing plant production capacity C i , Vehicle queue status QS i , Construction site demand D j , Received quantity RA j ; Action space: Generate a matching vector HLAV ∈ {0, 1,..., Ns × Nc - 1} between the mixing plant and the construction site, and the encoding rule is HLAV = station_idx × Nc + site_idx, where Ns represents the number of mixing plants, Nc represents the total number of construction sites, station_idx represents the index number of the mixing plant, and site_idx represents the index number of the construction site; Low-level decision-making: Input: time window TW, discrete time interval T i , available time slots; Action space: Generate a scheduling time vector LLAV ∈ {0, 1,..., TW / Ti - 1}, and the encoding rule is combined_action = HLAV × (TW / Ti) + LLAV.

4. The concrete dynamic scheduling method based on multi-level reinforcement learning and agent simulation according to claim 3, wherein Step S3 specifically includes: High-level DDQN: The input layer receives the state characteristics of the mixing plant and the construction site, the fully connected layer processes them, and the output layer outputs the Q-value matrix of the high-level DDQN; Low-level DDQN: The input layer receives the time window characteristics, the fully connected layer processes them, and the output layer outputs the Q-value matrix of the low-level DDQN; The training process includes: Experience replay: Store the state transition tuple (s t , a t , r t , s t +1) into the buffer, and sample in batches to update the network weights; Target network update: Synchronize the online network parameter θ to the target network θ' every set number of steps. The update formula is θ′←τθ+(1-τ)θ′, where s t represents the current state, including the production capacity C of the mixing plant i , the vehicle queue status QS i , the construction site demand D j , the transportation time TT ij , the in-transit concrete volume ITA ij information, and a t represents the current action, including the high-level action and the low-level action, and r t represents the immediate reward, calculated according to the reward calculation formula, and s t +1 represents the next state.

5. The concrete dynamic scheduling method based on multi-level reinforcement learning and agent simulation according to claim 4, characterized in that, The composite reward formula for reward calculation described in Step S3 is: R = αRcost + βRwait + γRdemand, where Rcost = -∑TC ij , Rwait = -∑WTC ij , Rdemand = ∑min(RA j , D j ); TC ij represents the transportation cost, α represents the weight of the transportation cost, β represents the weight of the waiting time cost, γ represents the weight of the demand satisfaction rate, and R represents the composite reward.

6. The concrete dynamic scheduling method based on multi-level reinforcement learning and agent simulation according to claim 5, characterized in that The dynamic adjustment method of weights α, β, γ is: According to the urgency of the construction site demand and the road congestion status, adopt an adaptive weight update formula: The same update β t+1 and γ t+1 , ensuring that the multi-objective optimization weights change dynamically with the environment.

7. The concrete dynamic scheduling method based on multi-level reinforcement learning and agent simulation according to claim 3, wherein Step S4 specifically includes: Event detection: Real-time monitoring of changes in construction site requirements ΔD j or road anomaly events; Rescheduling condition: If |ΔD j | or D j > 15% or the average vehicle waiting time is greater than the set time, then trigger the policy network to re-make a decision; Policy update: Based on the latest state s t Generate a new scheduling plan, and preferentially adjust the time slots and transportation routes of unexecuted tasks.

8. The concrete dynamic scheduling method based on multi-level reinforcement learning and agent simulation according to claim 1, characterized in that Step S2 includes the design of a double-layer fully connected feature extractor: A two-layer fully connected neural network is adopted, and the input layer receives a high-dimensional state vector After being processed by two hidden layers with 256 neurons, a low-dimensional feature vector \(f\in\mathbb{R}^{128}\) is output, which is used to capture the non-linear features in the production-transportation coupling relationship. The feature extractor is jointly trained with DDQN, and the weight matrices \(W_1\), \(W_2\) and the bias vectors \(b_1\), \(b_2\) are optimized through the loss function.

9. The concrete dynamic scheduling method based on multi-level reinforcement learning and agent simulation according to claim 1, wherein The process of integrating ABM and DDQN in the concrete dynamic scheduling mechanism in Step S4 includes: Set the initial vehicle position X i and speed V i (0) during the environment initialization phase; Every time the vehicle position X is updated i (t) and speed V i (t), and the updated traffic density ρ(t), average vehicle speed Congested section marker Fcongestion as the state input s t , for the DDQN to generate scheduling actions; Vehicle queue status QS i Updated in real time according to the number of tasks to be scheduled, and the formula is: QS i (t + 1)= QS i (t)- ∑ j SA ij (t)+ ∑ k δ ik (t), where δ ik (t) is the new task indication function, and SA ij (t) represents the total amount of concrete dispatched from the mixing plant to the construction site within time t.

Images