A multi-load agv double-layer reinforcement learning dynamic scheduling method fusing working condition perception

Abstract

The application discloses a multi-load AGV double-layer reinforcement learning dynamic scheduling method fusing working condition perception, and achieves the fusion of task allocation and path planning through problem formalization modeling, working condition feature extraction and dynamic weight adjustment, double-layer Markov decision process construction, task allocation based on working condition perception and double-layer D3QN, path planning based on integer programming, and two-stage collaborative scheduling, so that the multi-load AGV scheduling method capable of perceiving the working condition of a workshop and adaptively adjusting and optimizing the target is realized and provided.

Description

Technical Field

[0001] This invention belongs to the field of multi-load AGV scheduling technology in intelligent manufacturing systems, and particularly relates to a dynamic scheduling method for multi-load AGVs with dual-layer reinforcement learning that integrates working condition perception. Background Technology

[0002] With the rapid development of flexible manufacturing systems and smart factories, multi-load AGVs, with their ability to transport multiple materials at once, have become core equipment for improving the efficiency of material transportation in workshops. Compared to single-load AGVs, the scheduling of multi-load AGVs not only needs to consider task allocation but also optimize the access sequence of multiple task points. The quality of its scheduling scheme directly determines the transportation efficiency and on-time performance of the entire production system.

[0003] Existing multi-AGV scheduling systems are mostly designed for static task scenarios, assuming all task information is known in advance and solving them through analytical models and optimization algorithms. However, such methods are complex and difficult to solve when dealing with large-scale, complex systems, and cannot adapt to production environments with dynamically inserted tasks. For dynamic task scenarios, existing technologies mostly adopt methods based on a single scheduling rule, such as first-in-first-out, earliest delivery date, or shortest travel distance. However, a single scheduling rule performs differently under different production goals and system states, failing to maintain optimal performance under all operating conditions and lacking adaptability to complex dynamic environments. Furthermore, existing dynamic scheduling methods often use fixed optimization target weights, making it difficult to dynamically adjust the focus of the scheduling strategy according to the real-time production status of the workshop, resulting in a decline in scheduling performance when operating conditions change drastically.

[0004] Especially in multi-load AGV scheduling, the impact of new task points on the existing task sequence must be considered, making its state space and action space far more complex than that of a single-load AGV. Existing technologies separate task allocation and path planning into two independent stages, or use centralized modeling, resulting in excessively high solution complexity, making it difficult to meet the dual requirements of real-time performance and optimization for dynamic scheduling. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a dynamic scheduling method for multi-load AGVs that integrates working condition perception and two-layer reinforcement learning. Through formal problem modeling, working condition feature extraction and dynamic weight adjustment, construction of a two-layer Markov decision process, task allocation based on working condition perception and two-layer D3QN, path planning based on integer programming, and two-stage collaborative scheduling, the method achieves the integration of task allocation and path planning, and realizes and provides a multi-load AGV scheduling method that can perceive workshop working conditions and adaptively adjust and optimize objectives.

[0006] The technical solution of this invention is implemented as follows: A two-layer reinforcement learning dynamic scheduling method for multi-load AGVs that integrates operational condition awareness includes the following steps: S1. Establish a mathematical model for dynamic scheduling of multi-load AGVs, define an objective function, and the objective function includes dynamically adjustable weight factors; S2. Construct a working condition perception module to collect working condition characteristics in the production environment in real time, and adaptively generate dynamic weight factors in the objective function through a working condition-weight mapping network. S3. Decouple the task allocation process into two sub-problems: AGV scheduling rule selection and task selection rule selection. Construct a high-level Markov decision process model and a low-level Markov decision process model respectively, and define the state set, action set and reward function calculated based on dynamic weight factors for each model. S4. Construct a two-layer dual-depth Q-network architecture, in which the high-layer network takes the global state as input and outputs Q-values ​​of various AGV scheduling rules to determine the target AGV; the low-layer network takes the local state of the target AGV as input and outputs Q-values ​​of various task selection rules to determine the task allocation; the dynamic weight factor in the reward function is provided in real time by the working condition perception module. S5. During the online scheduling phase, based on the trained agent, the optimal AGV scheduling rules and task selection rules are selected according to the real-time status. S6. After assigning the selected task to the target AGV, the path replanning module based on integer programming is triggered to optimize the task point access order of the target AGV and generate a new path.

[0007] Furthermore, in S1, a dynamic scheduling mathematical model for multi-loaded AGVs is established, which specifically includes defining grid points in the production environment, dynamically inserting task attributes, and multi-loaded AGV operation constraints. The attributes of dynamically inserted tasks are described by a six-element vector, which includes the pickup point, delivery point, insertion time, delivery deadline, task availability, and task allocation status. The constraints for multi-load AGV operation include: each AGV has a fixed maximum capacity and travel speed, the number of tasks assigned to the same AGV does not exceed twice its capacity, and the pickup point for the same task must be visited before the delivery point.

[0008] Furthermore, in S1, the objective function is defined as minimizing the weighted average task delay and AGV travel distance. The objective function includes two adjustable dynamic weight factors, the sum of which is 1 and each of which takes a value within a preset range. Furthermore, in S2, the working condition characteristics in the production environment are collected in real time, including the proportion of task urgency, the overall load rate of AGV, the production cycle pressure and the AGV empty running rate. After the working condition characteristics are normalized, they are input into the working condition-weight mapping network, and the output is processed by normalization and boundary constraints to generate two dynamic weight factors at the current moment. Furthermore, in S2, the working condition sensing module includes: The working condition feature extraction unit is used to collect and calculate the normalized working condition feature vector; The load-weight mapping network is a lightweight fully connected neural network containing two hidden layers with 16 and 8 nodes respectively, and ReLU activation function. The output layer contains two nodes, corresponding to two dynamic weight factors, and Sigmoid activation function. The original output value is normalized by flexible maximum value and then scaled and translated to a preset range, while satisfying the constraint that the sum of the two weight factors is 1. The training unit, with the loss function being the minimization of the global scheduling objective function, is jointly trained with a two-layer duel dual-depth Q-network agent. Furthermore, in S3, the state set of the high-level Markov decision process model is formed by splicing the states of all AGVs and the states of all tasks. The AGV state includes the number of assigned tasks, the task points to be visited, and the estimated arrival time. The task state includes the pickup point, delivery point, remaining insertion time, remaining delivery deadline, availability, and allocation status. The state set of the low-level Markov decision process model consists of the state of the selected AGV and the state of all unassigned tasks; it focuses on the path information of the target AGV and the attributes of the tasks to be assigned, thereby improving the accuracy of task selection. Furthermore, in S3, the action set includes AGV scheduling rules and task selection rules; The AGV scheduling rules include: selecting the AGV with the fewest currently assigned tasks, selecting the AGV with the lowest driving efficiency, selecting the AGV with the shortest cumulative task delay time, selecting the AGV with the shortest remaining driving time, selecting the AGV with the shortest additional driving distance when inserting a new task, and selecting the AGV with the shortest distance between the current last task point and the nearest unassigned task pickup point. The task selection rules include: selecting the task with the earliest insertion time, selecting the task with the shortest distance between the pickup point and the delivery point, selecting the task with the earliest delivery deadline, selecting the task whose pickup point is closest to the selected AGV's current first task point, selecting the task that will result in the shortest additional travel distance for the selected AGV after being assigned to it, and selecting the task whose pickup point is closest to the selected AGV's current last task point. The reward function is calculated as follows: ; in, and The dynamic weighting factors generated for the operating condition sensing module Accumulated task delay time for AGVs. The AGV travel distance This represents the shortest average transport distance per task.

[0009] Furthermore, in S4, the network structure of the dual-layer duel dual-depth Q-network includes an input layer, three fully connected hidden layers, two parallel fully connected layers, and an output layer. The parallel fully connected layers output state value and action advantage, respectively, and generate action Q-value through a combination formula. The training process adopts an experience replay mechanism and an ε-greedy strategy, with the greedy parameter gradually decaying from its initial value to its minimum value as the training progresses. Every preset number of steps, the parameters of the main network are copied to the target network to reduce Q-value overestimation and stabilize training. Furthermore, in S5, the conditions for triggering path replanning during the online scheduling phase include: the completion of new task allocation, the update of the execution status of existing tasks, or changes in production environment constraints; each time it is triggered, the working condition perception module re-collects the current working condition characteristics and updates the dynamic weight factor to realize the closed-loop adaptation of the scheduling strategy to the real-time production status. Furthermore, in S6, the path replanning module based on integer programming is triggered, and constraints are set, including: each task point is visited only once, all task points are visited, and the pickup point is visited before the delivery point for the same task. The path replanning module for integer programming has the following objective function: ; in, and These represent the number of tasks that the AGV did not visit the pickup point and the number of tasks that were picked up but not delivered, respectively. To delay the task time, The time to reach the last mission point. The objective function represents the shortest distance between the pickup point and the delivery point; the weighting factors in this objective function are... and The same dynamic weighting factor as in S2 is used to keep the optimization objectives of the scheduling and path planning stages consistent.

[0010] When a new task is inserted or the task status is updated, S2-S6 are executed repeatedly to achieve adaptive dynamic scheduling. Compared with the prior art, the present invention achieves the following beneficial effects: 1. Implement adaptive adjustment of scheduling strategies to production conditions; By constructing a working condition sensing module, key working condition characteristics are collected in real time, and the working condition is utilized. The weighted mapping network generates dynamic weight factors in the objective function online, enabling the scheduling system to automatically adjust the optimization emphasis between task delay and travel distance based on the real-time status of the workshop. Compared with traditional methods using fixed weight factors, this invention effectively avoids the problem of scheduling performance degradation due to changes in operating conditions. It can reduce task-level weighted average delay under most typical operating conditions and demonstrates better adaptability to operating conditions such as emergency task outbreaks and distance pressure.

[0011] 2. The action space complexity of reinforcement learning is significantly reduced by decoupling through two-layer decision-making; The task allocation process is decoupled into two sub-problems: AGV scheduling rule selection and task selection rule selection. High-level and low-level Markov decision process models are constructed for each sub-problem. Furthermore, a two-layer duel dual-depth Q-network architecture is adopted. This hierarchical decision mechanism transforms the combined action space, which was originally related to the product of the number of AGVs and the number of tasks, into a fixed-size AGV scheduling rule space and task selection rule space. This significantly alleviates the problem of the action space rapidly expanding with the increase of the number of AGVs and tasks in the dynamic scheduling of multi-loaded AGVs, thereby improving the stability of model training.

[0012] 3. Possesses good real-time performance and generalization ability; The trained two-layer dual-depth Q-network only requires forward propagation to output the optimal rule during the online scheduling phase, resulting in low computational overhead and meeting the millisecond-level decision-making requirements in dynamic task insertion scenarios. Furthermore, since the network learns rules rather than specific AGVs or tasks, this method exhibits good transfer application potential and engineering scalability when the number of AGVs, tasks, or workshop layout changes. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a two-layer reinforcement learning dynamic scheduling method for multi-load AGVs that integrates working condition perception, provided in an embodiment of the present invention. Figure 2 This is a network architecture diagram of a condition-aware dual-layer D3QN for a multi-load AGV dual-layer reinforcement learning dynamic scheduling method provided in this embodiment of the invention. Figure 3 This is a comparison chart of the dynamic weighting method under different working conditions and the task-level weighted average delay without dynamic weighting baseline provided in the embodiments of the present invention, which is a dynamic scheduling method for multi-load AGV with integrated working condition perception and two-layer reinforcement learning. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0015] Example like Figures 1 to 3 A two-layer reinforcement learning dynamic scheduling method for multi-load AGVs that integrates working condition awareness includes the following steps: S1. Establish a mathematical model for dynamic scheduling of multi-load AGVs, define an objective function, and the objective function includes dynamically adjustable weight factors; Establish a mathematical model for dynamic scheduling of multi-loaded AGVs, specifically including defining grid points in the production environment, dynamically inserting task attributes, and multi-loaded AGV operation constraints; The attributes of dynamically inserted tasks are described by a six-element vector, which includes the pickup point, delivery point, insertion time, delivery deadline, task availability, and task allocation status. The constraints for multi-load AGV operation include: each AGV has a fixed maximum capacity and travel speed, the number of tasks assigned to the same AGV does not exceed twice its capacity, and the pickup point for the same task must be visited before the delivery point.

[0016] The objective function is defined as minimizing the weighted average task delay and AGV travel distance. The objective function includes two adjustable dynamic weight factors, the sum of which is 1 and each of which takes values ​​within a preset range.

[0017] In this embodiment, the dynamic scheduling problem of multi-load AGVs is defined as: in a system where multiple AGVs are dynamically scheduled... OK In a production environment composed of grid points, there exists a set of multi-load AGVs and a set of dynamically inserted transportation tasks. A grid point can be represented as... .

[0018] Each task It can be represented by a six-element vector Description, in which and These represent the pickup and delivery points for the task, respectively. and These represent the task insertion time and delivery deadline, respectively. and These are binary variables, representing whether the task is available and whether it has been assigned, respectively.

[0019] Equipped with a set of multi-load AGVs, denoted as Each AGV has the same maximum capacity. and driving speed An AGV can perform multiple tasks simultaneously, but the number of tasks assigned to the same AGV must not exceed its capacity. .

[0020] To evaluate the performance of the scheduling scheme, the objective function is constructed as follows, aiming to minimize the weighted average task delay and AGV travel distance: ; in, and As a weighting factor, ,and . This represents the total number of tasks inserted during the entire simulation cycle. The set of tasks for the entire simulation cycle. For the task The delay time, For AGV Total distance traveled during the entire cycle For the task The shortest distance between the pickup point and the delivery point.

[0021] This problem is solved collaboratively through two phases: task allocation and path planning. In the task allocation phase, the number of tasks assigned to the same AGV is limited to its capacity. To satisfy capacity constraints.

[0022] S2. Construct a working condition perception module to collect working condition characteristics in the production environment in real time, and adaptively generate dynamic weight factors in the objective function through a working condition-weight mapping network. The system collects real-time operating condition characteristics in the production environment, including the proportion of task urgency, the overall load rate of AGVs, the production cycle pressure, and the AGV empty running rate. After normalizing the operating condition characteristics, the system inputs them into the operating condition-weight mapping network. The output is then processed by normalization and boundary constraints to generate two dynamic weight factors for the current moment.

[0023] The operating condition sensing module includes: The working condition feature extraction unit is used to collect and calculate the normalized working condition feature vector; The load-weight mapping network is a lightweight fully connected neural network containing two hidden layers with 16 and 8 nodes respectively, and ReLU activation function. The output layer contains two nodes, corresponding to two dynamic weight factors, and Sigmoid activation function. The original output value is normalized by flexible maximum value and then scaled and translated to a preset range, while satisfying the constraint that the sum of the two weight factors is 1. The training unit, with the loss function being the minimization of the global scheduling objective function, is jointly trained with a two-layer duel dual-depth Q-network agent.

[0024] In this embodiment, to overcome the performance limitations of fixed weight factors under complex dynamic operating conditions, this method constructs a lightweight operating condition-weight mapping network to realize real-time dynamic adjustment of weight factors in the scheduling objective function.

[0025] First, to monitor the workshop production status in real time, four online-collectible operating condition features are extracted to form a normalized operating condition feature vector. This serves as the basis for dynamic weight adjustment: Task urgency percentage An urgent task is defined as one whose remaining delivery deadline does not exceed a preset percentage of the average remaining time of all tasks currently awaiting scheduling. ; in, ; ; Indicates the current rescheduling time. Indicates time The set of tasks to be scheduled in the system Number the task. Indicates task Delivery deadline, Indicates task At any moment The remaining delivery time, This represents the average remaining delivery time of the set of tasks to be scheduled. This represents the emergency task determination coefficient, with a preferred value of 0.5. This is an indicator function; it takes the value 1 if the condition within the parentheses is true, and 0 otherwise.

[0026] AGV overall load rate Defined as the ratio of the sum of the currently assigned tasks to all AGVs to the sum of the maximum task capacities of all AGVs, i.e.: ; in, Indicates time Already assigned to AGV And the number of tasks that have not yet been completed.

[0027] Production cycle pressure Defined as the ratio of the number of unfinished tasks to the maximum number of tasks that the system allows to exist simultaneously, it reflects the intensity of the workshop's production rhythm. ; in, Indicates time The set of tasks to be scheduled in the system This indicates the maximum number of tasks that the system allows to exist simultaneously.

[0028] AGV empty running rate Defined as the ratio of the sum of the empty driving distances of all AGVs to the sum of their total driving distances, i.e.: ; in, Indicates the time from the start of the scheduling period to time [time]. AGV The cumulative distance traveled without carrying any mission payload. Indicates AGV The cumulative distance traveled within the current scheduling cycle, It represents a very small positive number.

[0029] Based on the collected operating condition features, a lightweight fully connected neural network is constructed as an adaptive operating condition-weight regulator, i.e., an operating condition-weight mapping network. This network contains two hidden layers, with 16 and 8 nodes respectively, using ReLU activation. The output layer contains two nodes, corresponding to dynamic weight factors. , The activation function is Sigmoid, and normalization and boundary constraints are applied. The post-processing formula for the output layer is as follows: ; in, and The original network output, after Softmax normalization, is scaled to the range ([0.2, 0.8]) to ensure... And it satisfies the catch-all constraint. This is to avoid excessive weighting that could lead to scheduling imbalance.

[0030] Specifically, the training objective of the working condition-weight mapping network is to minimize the scheduling objective function. The global loss is defined as follows: ; in, For the load-weight mapping network parameters, For the experience replay pool, the loss function directly adopts the objective function with dynamic weights defined in S1. This allows the mapping network to share the same optimization objective with the subsequent two-layer D3QN network.

[0031] S3. Decouple the task allocation process into two sub-problems: AGV scheduling rule selection and task selection rule selection. Construct a high-level Markov decision process model and a low-level Markov decision process model respectively, and define the state set, action set and reward function calculated based on dynamic weight factors for each model. The state set of the high-level Markov decision process model is composed of the states of all AGVs and the states of all tasks. The AGV state includes the number of assigned tasks, the task points to be visited, and the estimated arrival time. The task state includes the pickup point, delivery point, remaining insertion time, remaining delivery deadline, availability, and allocation status. The state set of the low-level Markov decision process model consists of the state of the selected AGV and the state of all unassigned tasks; it focuses on the path information of the target AGV and the attributes of the tasks to be assigned, thereby improving the accuracy of task selection.

[0032] The action set includes AGV scheduling rules and task selection rules; The AGV scheduling rules include: selecting the AGV with the fewest currently assigned tasks, selecting the AGV with the lowest driving efficiency, selecting the AGV with the shortest cumulative task delay time, selecting the AGV with the shortest remaining driving time, selecting the AGV with the shortest additional driving distance when inserting a new task, and selecting the AGV with the shortest distance between the current last task point and the nearest unassigned task pickup point. The task selection rules include: selecting the task with the earliest insertion time, selecting the task with the shortest distance between the pickup point and the delivery point, selecting the task with the earliest delivery deadline, selecting the task whose pickup point is closest to the selected AGV's current first task point, selecting the task that will result in the shortest additional travel distance for the selected AGV after being assigned to it, and selecting the task whose pickup point is closest to the selected AGV's current last task point. The reward function is calculated as follows: ; in, and The dynamic weighting factors generated for the operating condition sensing module Accumulated task delay time for AGVs. The AGV travel distance This represents the shortest average transport distance for each task.

[0033] In this embodiment, to handle the complex action space in multi-load AGV scheduling, the task allocation process is decoupled into two sub-problems: AGV scheduling rule selection and task selection rule selection, and each is constructed as a Markov decision process model. This model can be defined as a six-tuple. ,in For a set of states, For action sets, As a discount factor, For the reward function, As a strategy.

[0034] First, define the state representation. To select the optimal rule under different environments, at time... Extract high-level states separately and lower-level state Both include AGV-related states and task-related states.

[0035] AGV The state is defined as ,in For AGV Number of tasks already assigned For the first One task point to be visited. This represents the estimated arrival time.

[0036] Task The state is defined as .

[0037] High-level status It is composed of the statuses of all AGVs and all tasks: .

[0038] Low-level state This includes the selected AGV. Status and the status of all tasks: .

[0039] Second, define the action space. High-level actions. For AGV scheduling rules, low-level actions Instead of directly selecting specific AGVs or tasks, rules are chosen for tasks to prevent the action space from exploding as the number of AGVs or tasks increases, thereby improving the model's generalization ability and learning efficiency.

[0040] AGV scheduling rules include 6 heuristic rules: Rule 1: Select the number of currently assigned tasks. Minimize the number of AGVs to balance the workload among them.

[0041] Rule 2: Select Driving Efficiency The lowest efficiency AGV. Travel efficiency is defined as the reciprocal of the average travel distance, as follows: ; ; ; in, Indicates that it has been assigned to AGV But not in time The set of tasks to be completed within the scope of the task. yes The sum of the shortest travel distances for the tasks. yes From time The total travel distance required to begin completing the task. Selecting the AGV with the lowest travel efficiency can lead to a re-optimization of the sequence, potentially improving its transport efficiency.

[0042] Rule 3: Select the current cumulative task delay time The smallest AGV. Task delay time is defined as follows: ; in, express Complete the task Time, This indicates the delivery deadline. Selecting the AGV with the shortest task delay time prioritizes ensuring the on-time performance of existing tasks and prevents overdue deliveries.

[0043] Rule 4: Select the AGV with the shortest remaining travel time.

[0044] Remaining driving time is defined as This rule allows for the fastest possible execution of new tasks without altering the existing task sequence.

[0045] Rule 5: Choose the additional travel distance required to perform the new task. The shortest AGV. The additional distance calculation method is as follows: Firstly in AGV Find the distance from the new task pickup point in the set of path points. and delivery points The two most recent points are: and .

[0046] Secondly, according to access and Calculate the additional formal distance in the order of the given information: ; choose The smallest AGV is selected to minimize the detour costs incurred by inserting new tasks.

[0047] Rule 6: Select the AGV with the shortest distance between the current last task location and the nearest unassigned task pickup location. Define AGV. The final mission point is Then its distance from the new task pickup point is This strategy of responding to nearby locations can reduce the AGV's idle time.

[0048] The task assignment rules include six heuristic rules: Rule 1: Select the insertion time The earliest task Follow the principle of first come, first served; Rule 2: Choose the distance between the pickup point and the delivery point Shortest task ; Rule 3: Select Delivery Period The earliest task; Rule 4: Select the task whose pickup point is closest to the first task point of the selected AGV, i.e.: Minimum task ; Rule 5: Select the task that minimizes the travel distance of the selected AGV after assignment. The calculation method is the same as AGV scheduling rule 5. ; Rule 6: Select the task whose pickup point is closest to the last task point currently being selected by the AGV, i.e.: Minimum task .

[0049] Finally, define the reward function. Used to assess at rescheduling points Execute low-level actions The subsequent impact on the objective function is calculated using the following formula: ; in, For the selected AGV, The shortest average transport distance for each task is used for normalization.

[0050] and The calculation was based on the new plan after task allocation and route replanning.

[0051] S4. Construct a two-layer dual-depth Q-network architecture, in which the high-layer network takes the global state as input and outputs Q-values ​​of various AGV scheduling rules to determine the target AGV; the low-layer network takes the local state of the target AGV as input and outputs Q-values ​​of various task selection rules to determine the task allocation; the dynamic weight factor in the reward function is provided in real time by the working condition perception module. The dual-layer Duel Dual-Depth Q-Network architecture includes an input layer, three fully connected hidden layers, two parallel fully connected layers, and an output layer. The parallel fully connected layers output state value and action advantage, respectively, and generate action Q-values ​​through a combination formula. The training process employs an experience replay mechanism and an ε-greedy strategy, where the greedy parameters gradually decay from their initial values ​​to their minimum values ​​as training progresses. Every preset number of steps, the parameters of the main network are copied to the target network to reduce Q-value overestimation and stabilize training.

[0052] In this embodiment, the two MDP models constructed by S3 are solved using the Dueling Double Deep Q-Network (D3QN) to achieve the optimal selection of AGV scheduling rules and task selection rules.

[0053] Both high-level D3QN and low-level D3QN adopt a duel network architecture, which uses the state value function. With action advantage function Separation, through formula The final Q value is generated by combining the results.

[0054] The two networks share the same basic structure: both contain an input layer, three fully connected hidden layers (with 512, 256, and 256 nodes respectively), followed by two parallel fully connected layers (each with 128 nodes), connecting the state value output layer and the action advantage output layer, respectively. However, their input dimensions, output spaces, and functional positioning differ fundamentally. High-level D3QN with As input, this state is composed of the multi-task point information of each AGV and the attribute states of all tasks, with a dimension of [missing information]. Its output layer consists of 6 nodes, corresponding to the Q-values ​​of 6 different AGV scheduling rules. Based on the overall workshop status, the most suitable AGV scheduling rule is selected, thereby determining the selected target AGV. .

[0055] Low-level D3QN with As input, this state focuses on the target AGV selected by the higher layer. Detailed information and attribute status of all unassigned tasks, with dimensions of Its output layer also has 6 nodes, corresponding to the Q-values ​​of the 6 task selection rules. The function of this network is: given a determined target AGV, to select the most suitable task selection rule, thereby determining the assigned task. .

[0056] The high-level D3QN and low-level D3QN collaborate sequentially in the decision-making process. First, the high-level network selects AGV rules and determines the target AGV based on the global state. Then, the low-level network selects task rules and determines the task assignment based on the local state of that AGV. Both share the same reward function during training. Achieve indirect collaboration and jointly complete the overall decision-making for multi-load AGV scheduling tasks.

[0057] Output the Q-values ​​for the six task selection rules, such as... Figure 2 As shown.

[0058] The training process employs a Dual Deep Q-Network (DDQN) mechanism, where the main network selects actions and the target network evaluates them to reduce Q-value overestimation. Simultaneously, an experience replay mechanism is used, storing the agent's experiences in an experience pool and randomly sampling in mini-batches during training to break down data correlations. Greedy strategy balances exploration and exploitation in the early stages of training. Larger values ​​are used to encourage exploration, and then gradually decreased to utilize learned knowledge. The training algorithms for high-level and low-level agents are as follows: Before training, the high-level experience pool is first initialized. and low-level experience pool The capacities are respectively and .

[0059] Initialize the high-level main network and its parameters High-level target network and its parameters ; Similarly, initialize the lower-level main network. and its parameters Low-level target network and its parameters .

[0060] Simultaneously, initialize the load condition-weight mapping network. and its parameters .

[0061] In each training round, the system initializes the environment. At each scheduling time... The system first collects current workshop operating condition data and calculates the normalized operating condition feature vector. Through a working condition-weight mapping network Generate real-time dynamic weight factors , .

[0062] Then, the system observes the current global high-level state. .

[0063] Subsequently, the higher-level action adopted an ε-greedy strategy: generating random numbers. ,like Then from the AGV scheduling rule set A rule is randomly selected uniformly from the given information; otherwise, the rule with the largest current Q value is selected, i.e.: ; According to the selected rule The system determines the specific AGV from the currently available AGV set. .

[0064] Subsequently, the system constructs a low-level state. This status focuses on AGVs The path information and the status of all incomplete tasks.

[0065] The low-level action selection also adopts the same approach. - Greedy strategy. Generate new random numbers. ,like Then from the set of task selection rules A rule is randomly selected uniformly from the given information; otherwise, the rule with the largest current Q value is selected, i.e.: ; Based on the selected task selection rules The system determines specific tasks from the current set of unassigned tasks. The system will assign tasks. Assigned to AGV And trigger the path planning module to process the AGV The task point access order is replanned to generate new paths. .

[0066] The system then calculates the instant reward based on the new path. And observe the high-level state at the next moment. and lower-level state .

[0067] To improve training stability, the algorithm employs an experience replay mechanism. After each interaction, high-level transfer samples are... Store in high-level experience pool Transfer samples from lower layers Store in the lower-level experience pool .

[0068] At the same time, the current working condition feature vector and the generated dynamic weighting factors , The transferred samples are stored together for use in subsequent training of the working condition-weight mapping network.

[0069] During training, mini-batches of samples are randomly sampled from two experience pools. For high-level networks, transfer samples are obtained through sampling. And use the target network to calculate the target Q value: ; Subsequently, the parameters of the high-level main network were analyzed. Perform gradient descent to minimize the mean squared error between the predicted Q-value and the target Q-value, i.e. .

[0070] Similarly, for low-level networks, sampling yields transferred samples. Calculate the target Q value: ; And parameters of the low-level main network Perform gradient descent.

[0071] Every The first step is to copy the main network parameters to the corresponding target network. , To stabilize the training process. During training, greedy parameters... Starting with an initial value of 0.1, the value gradually decreases to a minimum as the training progresses. This ensures that the agent fully explores the state space in the early stages of training and uses the learned knowledge to improve scheduling performance in the later stages of training.

[0072] This two-layer training framework decouples AGV scheduling from task selection and combines... The greedy strategy's exploration mechanism effectively reduces the dimensionality of the action space, improving the model's convergence speed and scheduling performance in complex dynamic environments.

[0073] S5. During the online scheduling phase, based on the trained agent, the optimal AGV scheduling rules and task selection rules are selected according to the real-time status. The conditions for triggering path replanning during the online scheduling phase include: the completion of new task allocation, the update of the execution status of existing tasks, or changes in production environment constraints. Each time it is triggered, the working condition perception module re-collects the current working condition characteristics and updates the dynamic weight factor to achieve closed-loop adaptation of the scheduling strategy to the real-time production status.

[0074] In this embodiment, after the high-level DQN and low-level DQN are trained, the resulting agent can be applied to a practical AGV scheduling system. Therefore, in practical applications, the model directly selects the action with the largest Q value, without needing to employ other methods. - A greedy strategy is used for random exploration, and the specific process is as follows: At each scheduling moment The system first observes the current global high-level state. This state contains path information for all AGVs and attributes for all incomplete tasks. Subsequently, the higher-level DQN uses the trained network parameters... Calculate the Q value of each AGV scheduling rule, and select the rule corresponding to the maximum value as the current optimal AGV scheduling rule.

[0075] According to the selected rule By definition, the system determines the specific AGV from the currently available AGV set. For example, if the selected rule is "select the AGV with the fewest tasks", the system will select the AGV with the currently assigned number of tasks from all available AGVs. The smallest AGV.

[0076] In AGV Once determined, the system constructs the low-level state. This state focuses on the selected AGV. The current path information and attributes of all unassigned tasks are then used. Subsequently, the lower-level DQN uses the network parameters from its training. Calculate the Q-value of each task selection rule and select the optimal rule: ; Based on the selected task selection rules The system determines specific tasks from the current set of unassigned tasks. For example, if the selected rule is "select the task with the earliest insertion time", the system will select the task with the smallest... The task.

[0077] The system will then assign the task Assigned to AGV And trigger the path planning module to process the AGV The access order is replanned to generate new paths. This path planning is based on an integer programming model, aiming to minimize the weighted sum of task delay and AGV travel distance, and optimizes the access order of multiple task points.

[0078] The execution framework runs in a loop until there are no more tasks to execute and no new tasks are inserted into the system. By directly selecting the action with the largest Q value, the trained DQN agent can quickly respond to task changes in dynamic environments, achieving efficient multi-load AGV scheduling.

[0079] S6. After assigning the selected task to the target AGV, the path replanning module based on integer programming is triggered to optimize the task point access order of the target AGV and generate a new path.

[0080] Trigger the route replanning module based on integer programming and set constraints, including: each task point is visited only once, all task points are visited, and the pickup point is visited before the delivery point for the same task. The path replanning module for integer programming has the following objective function: ; in, and These represent the number of tasks that the AGV did not visit the pickup point and the number of tasks that were picked up but not delivered, respectively. To delay the task time, The time to reach the last mission point. The objective function represents the shortest distance between the pickup point and the delivery point; the weighting factors in this objective function are... and The same dynamic weighting factor as in S2 is used to keep the optimization objectives of the scheduling and path planning stages consistent.

[0081] When a new task is inserted or the task status is updated, S2-S6 are executed repeatedly to achieve adaptive dynamic scheduling.

[0082] In this embodiment, after task allocation, the selected multi-load AGVs need to replan their task point access order to minimize the impact of new tasks on the execution efficiency of existing tasks. This method models the path planning problem as an integer programming model, optimizing the weighted average task delay and AGV travel distance while satisfying the task point access order constraint.

[0083] Let the current AGV The total number of assigned tasks is ,in This indicates the number of tasks that have not yet visited a pickup point. This represents the number of tasks that have been picked up but not yet delivered. All unvisited task points form a vector: ; in, This indicates the current position of the AGV.

[0084] Define binary decision variables , indicating task point Is it the first in the path? There are [number] points to visit. The objective function for path planning is: ; To ensure that each task point is visited exactly once, set the following constraints: ; ; To ensure that pickup points for the same task are visited before delivery points, set constraints: ; This integer programming model can be solved in real time using optimization solvers such as Gurobi. In dynamic task insertion scenarios, whenever a new task is assigned to an AGV, the system triggers path replanning to generate a new optimal access sequence. By collaborating with the DQN agent trained in S4, this two-stage framework achieves joint optimization of task allocation and path planning, effectively improving the overall operating efficiency of the multi-load AGV system while ensuring real-time performance.

[0085] This invention provides a two-layer reinforcement learning dynamic scheduling method for multi-load AGVs that integrates working condition perception. Specifically, it is used in flexible manufacturing workshops to achieve rapid response and efficient execution of dynamically inserted tasks through two-stage collaborative optimization of task allocation and path planning, thereby improving the intelligent scheduling capability and operating efficiency of the production system.

[0086] This invention constructs a closed-loop adaptive scheduling architecture consisting of "condition perception—dynamic weights—two-layer D3QN—path replanning". The condition perception module collects condition characteristics in real time, such as task urgency ratio, overall AGV load rate, production cycle pressure, and AGV empty run rate, and generates dynamic weight factors for the current moment through the condition-weight mapping unit. The generated dynamic weights are not only used in the online scheduling objective function and the path replanning objective function, but also directly participate in the reward function calculation during the two-layer D3QN training process, enabling the high-level AGV scheduling rule selection network and the low-level task selection rule selection network to learn differentiated scheduling strategies under different conditions.

[0087] To verify the adaptability of the dynamic weighted module for work condition perception under different production states, four typical work condition scenarios were set up in the simulation embodiment. The first scenario is a balanced work condition, where task arrival intervals are moderate, task delivery deadlines are relatively ample, and both production cycle pressure and empty-run rate are at a moderate level, used to simulate the normal operation of a workshop. The second scenario is an emergency task surge scenario, where task arrivals are more frequent, delivery deadlines are shorter, and production cycle pressure is higher, used to simulate delay-sensitive scenarios such as temporary order insertions and accelerated production pace. The third scenario is a distance pressure scenario, where task arrivals are relatively sparse, delivery deadlines are more lenient, but the AGV empty-run rate is high, used to simulate scenarios where tasks are spatially dispersed and the proportion of empty-load movement is increased. The fourth scenario is a mixed pressure scenario, where task arrivals are frequent and delivery deadlines are tight, while there is also some empty-run pressure, used to simulate complex work conditions where task timeliness and driving efficiency goals conflict simultaneously.

[0088] Figure 3The figure shows a comparison of the task-level weighted average delay between the dynamic weighting method and the baseline without dynamic weighting under different operating conditions. Experimental results show that the proposed dynamic weighting method reduces the task-level weighted average delay in balanced operating conditions, emergency task contingency conditions, and distance-pressure conditions, with improvements of 14.8%, 1.3%, and 40.4%, respectively; the average improvement across the four operating conditions is 12.75%. Specifically, the dynamic weighting factor is crucial in emergency task contingency conditions. , The values ​​of 0.662 and 0.338 demonstrate that the system can automatically increase the weighting of task delays; the dynamic weighting factor under distance-pressure conditions. , The values ​​of 0.488 and 0.512 demonstrate that the system can appropriately increase the distance-related weights based on empty driving and distance pressure. Under mixed pressure conditions, due to the simultaneous existence of task urgency and driving efficiency pressures, the dynamic weighting scheme... The increase is slight compared to the optimal fixed-weight baseline, reflecting the need for trade-offs in multi-objective conflict scenarios. Specifically, as follows... Figure 3 As shown.

[0089] It should be noted that under mixed pressure conditions, both task urgency and travel distance pressure coexist, making the conflict between scheduling objectives more pronounced. Experimental results show that the task-level weighted average latency of the dynamic weighting scheme in this scenario is slightly higher than the optimal fixed-weight baseline, indicating that when multiple pressure factors are superimposed, the dynamic weighting strategy may need to make more complex trade-offs between task timeliness and travel efficiency. Nevertheless, in the overall results across the four types of conditions, the task-level weighted average latency of the dynamic weighting scheme still achieved an average improvement of 12.75%, demonstrating that this module can improve the system's adaptability to real-time production status changes under most typical conditions. Overall, this invention can adaptively adjust the scheduling objective emphasis according to real-time conditions and improve the task-level weighted average latency under most typical conditions.

[0090] This invention focuses on key influencing factors in the dynamic scheduling of multi-load AGVs, defining four normalized work condition characteristics: task urgency ratio, overall AGV load rate, production cycle pressure, and AGV empty-run rate. The calculation methods for each characteristic are also provided. These characteristics characterize the real-time operating status of the workshop from aspects such as task timeliness, AGV task load, production rhythm intensity, and empty-run efficiency, giving the work condition perception results clear physical meaning and engineering interpretability; thus constructing a work condition characteristic system with clear physical interpretability.

[0091] This invention defines six AGV scheduling rules and six task selection rules. These rules cover multiple optimization principles such as load balancing, proximity response, priority for overdue tasks, and travel efficiency, and are all heuristic methods with extremely low computational complexity. By dynamically selecting the most suitable rule combination under different operating conditions through a two-layer D3QN, this invention retains the real-time advantage of rule-based methods while overcoming the poor adaptability of single fixed rules, achieving an organic combination of rule efficiency and learning adaptability; and constructing a comprehensive and engineering-applicable heuristic rule base.

[0092] This invention improves overall operational efficiency through the collaborative optimization of task allocation and path planning. Immediately after the agent completes task allocation, a path replanning module based on integer programming is triggered to precisely optimize the access order of multiple task points for the target AGV. Furthermore, the objective function for path planning also utilizes dynamic weighting factors provided by the condition perception module. This allows the task allocation and path planning phases to share the same optimization orientation, avoiding the internal friction problems caused by inconsistent weights or "delay considerations during allocation, but only distance considerations during planning" or other issues in traditional two-stage methods. Ultimately, it achieves a dynamic trade-off between task delay and travel distance, preventing inconsistencies in optimization objectives between the task allocation and path planning phases.

[0093] The trained two-layer D3QN model can be directly transferred to workshops of different sizes. Because the action space consists of a fixed number of heuristic rules, the model has good transfer application potential when the workshop task size or the number of AGVs varies within a certain range. Meanwhile, it is suitable for lightweight work environments. The weighted mapping network has very few parameters and fast inference speed, allowing it to run in real time on ordinary industrial control computers. These characteristics significantly reduce the deployment cost and technical threshold of this invention in practical flexible manufacturing systems. This invention has complete training capabilities. deploy The online adaptive closed-loop process enables the scheduling strategy to adapt to the production status in real time, which meets the requirements for the self-organization and self-adaptation capabilities of the smart workshop.

[0094] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the protection scope of the claims of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A two-layer reinforcement learning dynamic scheduling method for multi-load AGVs integrating working condition perception, characterized in that, Includes the following steps: S1. Establish a mathematical model for dynamic scheduling of multi-load AGVs, define an objective function, and the objective function includes dynamically adjustable weight factors; S2. Construct a working condition perception module to collect working condition characteristics in the production environment in real time, and adaptively generate dynamic weight factors in the objective function through a working condition-weight mapping network. In S2, the working condition characteristics in the production environment are collected in real time, including the proportion of task urgency, the overall load rate of AGV, the production cycle pressure and the AGV empty running rate. After the working condition characteristics are normalized, they are input into the working condition-weight mapping network to generate two dynamic weight factors at the current moment. In S2, the working condition sensing module includes: The working condition feature extraction unit is used to collect and calculate the normalized working condition feature vector; The load-weight mapping network is a lightweight fully connected neural network containing two hidden layers with 16 and 8 nodes respectively, and ReLU activation function. The output layer contains two nodes, corresponding to two dynamic weight factors, and Sigmoid activation function. The original output value is normalized by flexible maximum value and then scaled and translated to a preset range, while satisfying the constraint that the sum of the two weight factors is 1. The training unit, with the loss function being the minimization of the global scheduling objective function, is jointly trained with a two-layer duel dual-depth Q-network agent. S3. Decouple the task allocation process into two sub-problems: AGV scheduling rule selection and task selection rule selection. Construct a high-level Markov decision process model and a low-level Markov decision process model respectively, and define the state set, action set and reward function calculated based on dynamic weight factors for each model. S4. Construct a two-layer dual-depth Q-network architecture, in which the high-layer network takes the global state as input and outputs Q-values ​​of various AGV scheduling rules to determine the target AGV; the low-layer network takes the local state of the target AGV as input and outputs Q-values ​​of various task selection rules to determine the task allocation; the dynamic weight factor in the reward function is provided in real time by the working condition perception module. S5. During the online scheduling phase, based on the trained agent, the optimal AGV scheduling rules and task selection rules are selected according to the real-time status. S6. After assigning the selected task to the target AGV, the path replanning module based on integer programming is triggered to optimize the task point access order of the target AGV and generate a new path. In S6, the conditions for triggering path replanning during the online scheduling phase include: the completion of new task allocation, the update of the execution status of existing tasks, or changes in production environment constraints. Each time it is triggered, the working condition perception module re-collects the current working condition characteristics and updates the dynamic weight factor to realize the closed-loop adaptation of the scheduling strategy to the real-time production status.

2. The method for dynamic scheduling of multi-load AGVs with integrated working condition perception through two-layer reinforcement learning according to claim 1, characterized in that, In S1, a dynamic scheduling mathematical model for multi-loaded AGVs is established, which specifically includes defining grid points in the production environment, dynamically inserting task attributes, and multi-loaded AGV operation constraints. The attributes of dynamically inserted tasks are described by a six-element vector, which includes the pickup point, delivery point, insertion time, delivery deadline, task availability, and task allocation status. The constraints for multi-load AGV operation include: each AGV has a fixed maximum capacity and travel speed, the number of tasks assigned to the same AGV does not exceed twice its capacity, and the pickup point for the same task must be visited before the delivery point.

3. The method for dynamic scheduling of multi-load AGVs with integrated working condition perception through two-layer reinforcement learning according to claim 1, characterized in that, In S1, the objective function is defined as minimizing the weighted average task delay and AGV travel distance. The objective function includes two adjustable dynamic weight factors, the sum of which is 1 and each of which takes a value within a preset range.

4. The method for dynamic scheduling of multi-load AGVs with integrated working condition perception through two-layer reinforcement learning according to claim 1, characterized in that, In S3, the state set of the high-level Markov decision process model is formed by splicing the state of all AGVs and the state of all tasks. The AGV state includes the number of assigned tasks, the task points to be visited, and the estimated arrival time. The task state includes the pickup point, delivery point, remaining insertion time, remaining delivery deadline, availability, and allocation status. The state set of a low-level Markov decision process model consists of the state of the selected AGV and the states of all unassigned tasks.

5. The method for dynamic scheduling of multi-load AGVs with integrated working condition perception through two-layer reinforcement learning according to claim 1, characterized in that, In S3, the action set includes AGV scheduling rules and task selection rules; The AGV scheduling rules include: selecting the AGV with the fewest currently assigned tasks, selecting the AGV with the lowest driving efficiency, selecting the AGV with the shortest cumulative task delay time, selecting the AGV with the shortest remaining driving time, selecting the AGV with the shortest additional driving distance when inserting a new task, and selecting the AGV with the shortest distance between the current last task point and the nearest unassigned task pickup point. Task selection rules include: selecting the task with the earliest insertion time, selecting the task with the shortest distance between the pickup point and the delivery point, selecting the task with the earliest delivery deadline, selecting the task whose pickup point is closest to the selected AGV's current first task point, selecting the task that, when assigned to the selected AGV, results in the shortest additional travel distance for that AGV, and selecting the task whose pickup point is closest to the selected AGV's current last task point. The reward function is calculated as follows: ； in, and The dynamic weighting factors generated for the operating condition sensing module Accumulated task delay time for AGVs. The AGV travel distance Let t represent the shortest average transport distance for each task, t+1 represent the next time step in reinforcement learning, and t+1 represent the next time step in reinforcement learning.

6. The method for dynamic scheduling of multi-load AGVs with integrated working condition perception through two-layer reinforcement learning according to claim 1, characterized in that, In S4, the network structure of the dual-layer duel dual-depth Q-network includes an input layer, three fully connected hidden layers, two parallel fully connected layers, and an output layer. The parallel fully connected layers output state value and action advantage, respectively, and generate action Q-value through a combination formula. The training process adopts an experience replay mechanism and an ε-greedy strategy. The greedy parameter gradually decays from its initial value to its minimum value as the training progresses. Every preset number of steps, the parameters of the main network are copied to the target network to reduce Q-value overestimation and stabilize training.

7. The method for dynamic scheduling of multi-load AGVs with integrated working condition perception through two-layer reinforcement learning according to claim 1, characterized in that, In step S6, the path replanning module based on integer programming is triggered, and constraints are set, including: each task point is visited only once, all task points are visited, and the pickup point is visited before the delivery point for the same task. The path replanning module for integer programming has the following objective function: ； in, and These represent the number of tasks that the AGV did not visit the pickup point and the number of tasks that were picked up but not delivered, respectively. To delay the task time, The time to reach the last mission point. The shortest distance between the pickup point and the delivery point; weighting factors in the objective function. and The same dynamic weighting factor as in S2 is used to keep the optimization objectives of the scheduling and path planning stages consistent. When a new task is inserted or the task status is updated, S2-S6 are executed repeatedly to achieve adaptive dynamic scheduling.

Images