Discrete manufacturing agv scheduling system and method fusing digital twin

Abstract

The application discloses a kind of discrete manufacturing AGV scheduling systems and methods of fusion digital twin, including flexible job digital twin workshop system, job task scheduling agent and AGV scheduling agent;It combines deep Q network reinforcement learning algorithm with high-fidelity digital twin workshop simulation environment, comprehensively considers production job task scheduling, establishes kinematics constraint for AGV based on workshop real layout, production unit and process flow on digital twin platform, real-time extraction task state, equipment load, AGV operating state and other multidimensional features, constructs the deep reinforcement learning model including state space, action space and reward function, uses DQN algorithm for multi-round task scheduling strategy training in digital twin environment, evaluates scheduling scheme by virtual deduction, feedback optimization strategy parameter, and maps optimal scheduling scheme to physical workshop, solve the problem that theoretical scheduling and actual application cannot be synchronized in existing AGV scheduling algorithm, and the problem of insufficient adaptability.

Description

Technical Field

[0001] This invention relates to the fields of intelligent manufacturing and industrial automation, and is particularly applicable to discrete manufacturing AGV scheduling systems and methods that integrate digital twins. Background Technology

[0002] Manufacturing enterprises face immense pressure from fierce market competition and rising manufacturing costs. Therefore, continuously improving production efficiency and reducing costs has always been a key focus for manufacturing companies. Driven by recent advancements in the Internet of Things (IoT) and Industrial Artificial Intelligence (AI), many information technologies (RFID, embedded devices, augmented reality, etc.) and industrial robots (robotic arms, mobile robots, etc.) have been widely applied in production workshops. Automated Guided Vehicles (AGVs), mobile robots used for material handling, are considered one of the most promising technologies due to their high degree of autonomy and flexibility, and have been applied to material supply operations in various workshop and warehouse logistics.

[0003] There are two key limitations in the scheduling problem of AGVs between theoretical research and industrial applications. One limitation is that scheduling and path planning models simplify the kinematic constraints of AGVs, such as assuming that the speed of AGVs is fixed and constant. This fails to meet the spatiotemporal constraints and kinematics of AGVs, leading to a gap between theoretical scheduling and practical applications, resulting in local congestion or deadlock. On the other hand, while exact algorithms, heuristic rules, and metaheuristic algorithms can arrive at optimal or feasible solutions in certain specific environments, balancing performance, responsiveness, and adaptability remains challenging.

[0004] Currently, reinforcement learning (RL) and deep reinforcement learning (DRL) AGV scheduling methods exhibit excellent generalization capabilities in complex environments. However, DRL algorithms belong to self-supervised trial-and-error learning methods, requiring a data-rich and secure simulation environment to support the training of DRL-based agents. Summary of the Invention

[0005] The purpose of this invention is to provide a discrete manufacturing AGV scheduling system and method that integrates digital twins. By combining with a digital twin system, it provides a rich and secure simulation environment for AGV scheduling algorithms, reduces training costs and improves adaptability, and solves the problems of the inability to synchronize theoretical scheduling with practical applications and insufficient adaptability in existing AGV scheduling algorithms.

[0006] To achieve the above objectives, the discrete manufacturing AGV scheduling system integrating digital twins described in this invention includes a flexible operation digital twin workshop system, a task scheduling intelligent agent, and an AGV scheduling intelligent agent; The flexible operation digital twin workshop system integrates the layout, equipment parameters and production process of the physical workshop, assigns production control logic to all equipment and simulates the production status of the physical workshop in real time, initiates AGV scheduling decisions based on decision points, and performs and issues scheduling decisions. The task scheduling agent, based on the task characteristics and equipment status characteristics provided by the flexible operation digital twin workshop system, adopts a solution model based on deep Q-networks and outputs a task scheduling strategy, including the priority and execution sequence of tasks, through state space, action space and reward function. The AGV scheduling agent, based on the AGV state characteristics provided by the flexible operation digital twin workshop system, outputs a task scheduling strategy. It adopts a solution model based on a deep Q-network and outputs an AGV allocation strategy through the state space, action space, and reward function.

[0007] Furthermore, the task scheduling agent and the AGV scheduling agent initiate online fine-tuning based on the deduction results of the scheduling decisions by the flexible operation digital twin workshop system; the deduction results include task delay rates exceeding thresholds and equipment utilization rates falling below thresholds; the online fine-tuning adopts a phased optimization strategy, placing the deduction trajectory of the current scheduling decision as an independent sample into the experience replay buffer; a combination of random sampling and importance sampling is used to batch extract sample data from the experience replay buffer, calculate the loss function of the current depth Q network relative to the target depth Q network, use an adaptive moment estimation optimizer to perform gradient descent updates, and simultaneously set a gradient pruning threshold to prevent gradient explosion, until the current depth Q network converges; the target depth Q network updates its network parameters with a fixed time constant.

[0008] Furthermore, the task scheduling agent comprises an input layer, an output layer, and five hidden layers; the number of nodes in the input and output layers is the same as the number of nodes in the state space and action space, respectively; each hidden layer contains 30 nodes; the input and hidden layers use the tansig activation function, and the output layer uses the purelin activation function; the state space of the task scheduling agent includes average equipment utilization, standard deviation of average equipment utilization, average operation completion rate, average task completion rate, standard deviation of average task completion rate, estimated task delay rate, and actual task delay rate; the action space of the task scheduling agent includes six rules for determining task priorities and execution sequences; the reward function of the task scheduling agent consists of the actual task delay rate, the estimated task delay rate, and the average equipment utilization.

[0009] Furthermore, the AGV scheduling agent comprises an input layer, three hidden layers, and an output layer; the number of nodes in the input and output layers is the same as the number of nodes in the state space and action space, respectively; the three hidden layers have decreasing numbers of neurons; all hidden layers use the ReLU activation function; the input and output layers do not use activation functions; the state space of the AGV scheduling agent includes the average AGV utilization rate, the standard deviation of the AGV utilization rate, the average AGV completion time, and the standard deviation of the average AGV completion time; the action space of the AGV scheduling agent includes all available AGVs in the discrete manufacturing workshop AGV scheduling system integrating digital twins; the reward function of the AGV scheduling agent is the weighted sum of the change in the maximum completion time of AGVs and the change in the average utilization rate of AGVs.

[0010] Furthermore, the decision point includes when a new job task arrives at the flexible digital twin workshop system or when a job task operation is completed.

[0011] Furthermore, the task characteristics include decision points. Time, assignment Number of operations completed Decision point Time, assignment completion rate The device status characteristics include decision points. At any time, equipment Completion time of the last task operation Decision point At any time, equipment average output rate The AGV status characteristics include the AGV trolley. utilization rate Decision point At any moment, the AGV cart The sum of total transmission time and waiting time .

[0012] Furthermore, the scheduling decision simulation employs time compression technology, simulating production until the next decision point emerges based on the task scheduling strategy and AGV allocation strategy determined by the task scheduling agent and AGV scheduling agent.

[0013] Furthermore, once the scheduling decision simulation results achieve the expected goals, the scheduling decision is transmitted to the execution system of the physical workshop through scheduling decision instruction parsing and standardized industrial communication interface.

[0014] The discrete manufacturing workshop AGV scheduling method integrating digital twins as described in this invention, based on the discrete manufacturing workshop AGV scheduling system integrating digital twins, includes the following steps: S1 constructs a flexible digital twin workshop system based on the layout of the physical workshop, equipment parameters, and production process flow, and endows all equipment with production control logic and precise geometric attributes; S2, based on the deep Q-network solution model, defines a state space with seven state features and an action space with six features to determine the priority and execution sequence rules of the tasks. The task scheduling agent is constructed with the task delay rate, the estimated task delay rate and the average equipment utilization rate as the core reward functions. S3, based on the deep Q-network solution model, defines a state space with four state features, which includes the action space of all available AGVs. The reward function is the weighted sum of the change in the maximum completion time of AGV and the change in the average utilization rate of AGV, and an AGV scheduling agent is constructed. S4, configure the training parameters for the job scheduling agent and the AGV scheduling agent respectively, including the number of training cycles, batch size, discount factor, target network update parameters, exploration rate decay strategy, experience replay buffer size, learning rate, optimizer parameters and network structure parameters; S5 defines decision points and extracts multi-dimensional feature parameters in real time from the flexible operation digital twin workshop system, including operation task features, equipment status features and AGV status features; S6, start simulation training; the job task scheduling agent receives job task characteristics and equipment status characteristics and outputs job task scheduling strategy; the AGV scheduling agent receives AGV status characteristics and the output of the job task scheduling agent to obtain AGV allocation strategy. S7, the flexible operation digital twin workshop system accelerates the simulation based on the task scheduling strategy and AGV allocation strategy, and monitors key performance indicators in real time, including task delay rate and equipment utilization rate. S8, determine whether the key performance indicators meet expectations; if not, start online fine-tuning; store the simulation trajectory in the experience playback buffer, and update the network parameters of the job scheduling agent and the AGV scheduling agent through gradient descent; S9, if the expected results are achieved, the task scheduling strategy and AGV allocation strategy are parsed into control instructions for specific equipment and sent to the physical workshop execution system through a standardized industrial communication interface.

[0015] Furthermore, the control instructions for specific equipment include AGV path coordinates, operating parameters, and machine tool processing programs.

[0016] Compared with existing technologies, this invention combines the Deep Q-Network (DQN) reinforcement learning algorithm with a high-fidelity digital twin workshop simulation environment, providing a verifiable and secure training and verification environment for Deep Q-Network reinforcement learning. This invention comprehensively considers production task scheduling, taking AGV scheduling in a discrete manufacturing workshop scenario as the research object. Based on the actual workshop layout, production units, and process flow on the digital twin platform, kinematic constraints are established for the AGVs. Multi-dimensional features such as task status, equipment load, and AGV operating status are extracted in real time from the digital twin platform to construct a deep reinforcement learning model including state space, action space, and reward function. The DQN algorithm is used to train the task scheduling strategy in multiple rounds within the digital twin environment. The scheduling scheme is evaluated through virtual simulation, and the optimal strategy parameters are fed back for optimization. The optimal scheduling scheme is then mapped to the physical workshop, solving the problems of asynchronous theoretical scheduling and practical application, and insufficient adaptability in existing AGV scheduling algorithms. This provides a rich and secure simulation environment for AGV scheduling algorithms and reduces training costs. Attached Figure Description

[0018] Figure 1 This is a system architecture diagram of the present invention.

[0019] Figure 2 This is a flowchart of the method described in this invention. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] The technical terms used in this invention are explained as follows: Digital twins can achieve high-fidelity modeling of physical workshops through virtual-real mapping, providing a verifiable and secure training and verification environment for reinforcement learning.

[0022] AGVs, or Automated Guided Vehicles, are devices in Industrial Logistics Systems (IPLS) responsible for horizontal material transport between racks and machines, executing material handling tasks sequentially. IPLS also includes robots, stacker cranes, manufacturing cells, and racks. Typically, an AGV moves from its current location to a loading point, picks up materials, and transports them to an unloading point. Robots play a crucial role in material handling between the AGV and buffer zones, while stacker cranes handle material storage and retrieval operations within the warehouse. Therefore, the scheduling of AGVs in discrete manufacturing workshops in industrial settings is of paramount importance.

[0023] The purpose of this invention is to address the task scheduling problem of AGVs in discrete workshops. By combining task priorities and using a digital twin platform for strategy deduction and verification, a scheduling mapping mechanism of "virtual decision-making - real-world guidance" is constructed. This provides a new perspective for data-driven decision-making and optimization in dynamic environments and enhances the self-learning, self-optimization and adaptive capabilities of manufacturing systems.

[0024] like Figure 1 As shown, the discrete manufacturing AGV scheduling system integrating digital twins described in this invention includes a flexible operation digital twin workshop system, an operation task scheduling intelligent agent, and an AGV scheduling intelligent agent.

[0025] The flexible operation digital twin workshop system integrates the layout, equipment parameters, and production process of the physical workshop, assigns production control logic to all equipment, simulates the production status of the physical workshop in real time, initiates AGV scheduling decisions based on decision points, and performs and issues scheduling decisions.

[0026] The flexible operations digital twin workshop system is built in Unity, utilizing 3D modeling tools to create precise geometric models of equipment such as workshops, shelves, manufacturing cells, robots, and stacker cranes. Mesh models and skeletal animations, including details such as dimensions and wheels, are created for the AGVs. Production control logic is assigned to all equipment and motion logic to the AGVs using C# scripts, with strict definitions of kinematic constraints such as linear velocity, angular velocity, position, and attitude. The NavMesh autonomous pathfinding algorithm is used to achieve path planning and obstacle avoidance from loading to unloading points. To facilitate subsequent parameter feature extraction, JSON files are used to store multi-dimensional features such as task status, equipment load, and AGV operating status. A data interface is developed to communicate with external scheduling algorithms (such as Python reinforcement learning models) to receive task instructions and synchronize equipment status. Specifically, In this invention, a flexible digital twin workshop system is used to simulate a discretized, flexible production workshop, where n consecutively arriving work tasks J = {J1, J2, ..., J...} n} With m devices M={M1, M2, ..., M m The above processing has a total of h usable AGV vehicles, V = {V1, V2, ..., V}. h} Perform horizontal material transport. Each work task J i Contains n i One operation; Job task J i The j-th operation O i,j Compatible with compatible devices M i,j (M) i,j Choose any one device from ⊆M) Execution. The total delay time for all tasks is minimized through AGV scheduling. Based on actual production conditions, the simulation of the flexible operations digital twin workshop system also includes the following constraints: (1) Each machine can only process one operation at a time (i.e., capacity limit).

[0027] (2) All operations in the same job must be executed in a fixed order (i.e., priority constraint).

[0028] (3) Each operation must be processed continuously and cannot be interrupted.

[0029] (4) The completion time of each operation must be a non-negative number.

[0030] (5) Each operation can select multiple machines, but can only be assigned to one machine in the end.

[0031] (6) Tasks can only be processed after their arrival time.

[0032] At the same time, the task J i The j-th operation O i,j The processing time is denoted as t. i,j,k Homework Task J i The j-th operation O i,j The actual completion time is denoted as C. i,j Homework Task J i The arrival time at the workshop and the completion deadline are respectively denoted as A. i and D i .

[0033] It is particularly important to note that the physical properties of the AGV itself, including but not limited to the precise outline and dimensions of the vehicle body, and the specific specifications and layout of the wheels, directly determine the effectiveness of the simulation of the flexible operation digital twin workshop system. When constructing a flexible operation digital twin workshop system, it is necessary to perform high-fidelity geometric modeling of the AGV, accurately reproduce its physical properties, and determine the AGV's passable areas, minimum turning radius, and maneuverability in narrow passages in both virtual and real environments. This is to avoid the scheduling algorithm theoretically obtaining the optimal solution, but then causing system paralysis due to virtual collision misjudgments or actual physical interference.

[0034] The motion logic of AGV needs to comprehensively consider the linear velocity, angular velocity, position and attitude of AGV, which can be expressed by formula (1) as follows: (1) in, This refers to AGV numbered j. , , They are At the present moment of coordinate, Coordinates and orientation angle They are In the next moment of coordinate, Coordinates and orientation angle These are the linear velocity and angular velocity of AGVj at the current time t. It is the time step.

[0035] The flexible operation digital twin workshop system of this invention integrates the proportional-integral-derivative (PID) control method, as shown in formulas (2) to (5).

[0036] (2) (3) (4) (5) in, It is the velocity vector of the AGV numbered j at time t; It is the running speed set for obstacle avoidance by the AGV numbered j at time t. This is the actual speed of AGV j at time t+1, obtained through the PID algorithm. Kp, Ki, and Kd represent the proportional, integral, and derivative time constants, respectively, with specific values ​​such as Kp=0.2, Ki=0.1, and Kd=0.02.

[0037] The task scheduling agent, based on the task characteristics and equipment status characteristics provided by the flexible operation digital twin workshop system, adopts a solution model based on deep Q-networks to determine the priority and execution sequence of tasks through state space, action space, and reward function.

[0038] In the discrete manufacturing AGV scheduling system integrating digital twins described in this invention, feature parameters serve as a bridge connecting the digital twin environment and the decision-making agent. Real-time capture and quantification of key indicators that comprehensively and accurately reflect the instantaneous state and dynamic evolution trend of the production system from a high-fidelity flexible digital twin workshop system enables the subsequent construction of a high-dimensional, structured, and information-rich state space for the decision-making agent. The completeness and representational capability of this state space directly determine the depth and breadth of the decision-making agent's understanding of the complex production environment, thereby affecting its efficiency in learning the optimal scheduling strategy and its final performance.

[0039] The task scheduling agent relies on task characteristics and equipment status characteristics provided by the flexible operations digital twin workshop system. Task characteristics describe order progress, while equipment status characteristics reflect the tightness of processing resources. Accurate extraction of task characteristics and equipment status characteristics aims to provide quantitative data on production schedule pressure and manufacturing resource bottlenecks for the discrete manufacturing AGV scheduling system integrating digital twins.

[0040] The characteristics of the task include decision points. Time, assignment Number of operations completed Decision point Time, assignment completion rate . It is an absolute indicator that directly measures the progress of a single task. Used to describe each job task J i The completion percentage is the core basis for the discrete manufacturing AGV scheduling system integrating digital twins described in this invention to intelligently judge the urgency of tasks, identify potential delay risks, and dynamically allocate priorities among different tasks. Through decision points Time, assignment Number of operations completed Compared to the above homework tasks Total number of operations n i The conclusion is as follows.

[0041] The device status characteristics include decision points. At any time, equipment Completion time of the last task operation Decision point At any time, equipment average utilization rate . It is not a simple record of the current time, but a predicted value based on the current queue and operation duration, which dynamically identifies the device. The "promised delivery point" on the future timeline is the evaluation equipment. Key to future availability. It measures the equipment per unit time. Average output rate, or machine M k Effective utilization rate of time resources, through decision points At any time, equipment The number of operations completed on the device is less than the number of operations completed on the device. Completion time of the last task operation Obtained. A sustained high The value not only indicates the equipment This is the current production bottleneck, and it also indicates that the equipment... Insufficient flexibility in responding to unexpected tasks; and too low The value indicates the device Resource idleness and declining return on investment.

[0042] The task scheduling agent is used to determine the priority and execution sequence of tasks. It adopts a deep Q-network (DQN) solution model to transform the task priority and execution sequence problem into a Markov decision process, defining the state space, action space and reward function.

[0043] 1. Define the state space This invention designs seven state features, each with a value range limited to the [0,1] interval, which serve as input parameters for DQN. By restricting all state features to the [0,1] range, the job scheduling agent of this invention can be easily extended to different untrained production environments. The seven state features are as follows: (1) Average equipment utilization rate As shown in formula (6): (6) (2) Standard deviation of average equipment utilization As shown in the formula: (7) (3) Average operation completion rate As shown in formula (8): (8) (4) Average task completion rate As shown in formula (9): (9) (5) Standard deviation of average task completion rate As shown in formula (10): (10) (6) Estimated task delay rate ; Will Defined as the average completion time of the last batch of jobs on all devices at time t. At this point, the estimated delayed jobs refer to those with remaining processing time exceeding the time since the last batch of jobs. Tasks with free time until the deadline. Estimated delay. It is defined as the estimated number of delayed tasks divided by the total number of unfinished tasks among all remaining tasks.

[0044] (7) Actual task delay rate ; Actual delayed tasks refer to tasks that were not completed before the deadline. Actual task delay rate. Defined as the ratio of the actual number of delayed tasks to the total number of unfinished tasks among all remaining tasks.

[0045] 2. Define the action space The discrete manufacturing AGV scheduling system integrating digital twins described in this invention requires not only formulating task selection rules at each decision node, but also determining equipment allocation rules. That is, it needs to solve two sub-problems: task sequence planning and AGV allocation. Based on this, this invention defines six rules for the task scheduling agent to determine task priorities and execution sequences, as follows: Rule 1 (Action 1): If there are no delayed tasks at the current time, sort the remaining tasks by their average idle time and select the task with the shortest idle time as the next operation. If there are delayed tasks at the current time, select the task with the largest delay as the next operation. After determining the tasks to be processed, assign them to the earliest available device. If multiple devices have the same earliest available time, randomly select one for assignment.

[0046] Rule 2 (i.e., Action 2): If there are no delayed tasks at the current time, sort them according to the ratio of task slack time to remaining processing time, and select the task with the smallest ratio to proceed to the next operation. If there are delayed tasks at the current time, select the task with the largest delay as the next operation. After determining the tasks to be processed, assign them to the earliest available device.

[0047] Rule 3 (i.e. Action 3): First, select the task with the longest expected delay time for the next step, and then assign the selected task to the equipment with the lowest utilization rate with a probability of 0.5; if the probability is less than 0.5, assign it to the equipment with the lowest workload.

[0048] Rule 4 (i.e. Action 4): Randomly select an unfinished operation of a job task and assign it to the earliest available device.

[0049] Rule 5 (i.e., Action 5): If there are no delayed tasks at the current time, sort them according to the product of the completion rate and slack time for each task, and select the task with the smallest product as the next operation. If there are delayed tasks at the current time, sort them according to the product of the inverse completion rate and the estimated delay time for each task, and select the task with the largest product as the next operation. The next operation for the finally selected task will be assigned to the earliest available device.

[0050] Rule 6 (i.e. Action 6): Select the next action of the task with the longest estimated delay in the current task, and then assign the action to the device with the earliest available time.

[0051] 3. Define the reward function The reward function of the task scheduling agent is based on the actual task delay rate, the estimated task delay rate, and the average equipment utilization rate.

[0052] Finally, the task scheduling agent employs a Deep Q-Network (DQN model) consisting of one input layer, one output layer, and five hidden layers. The number of nodes in the input and output layers is the same as the number of nodes in the state and action spaces, respectively; each hidden layer contains 30 nodes; the input and hidden layers use the tansig activation function, while the output layer uses the purelin activation function.

[0053] The AGV scheduling agent determines which AGV will execute the task sequence determined by the task scheduling agent. The AGV scheduling agent obtains AGV state characteristics from the flexible operation digital twin workshop system, and based on the task priority and execution sequence decided by the task scheduling agent, uses a deep Q-network-based solution model to determine the AGV allocation scheme through the state space, action space, and reward function.

[0054] Firstly, as a logistics link connecting various manufacturing units, the goal of extracting AGV operational characteristics is to quantify the workload, service efficiency, and real-time availability of AGVs, thereby providing data support for the precise allocation of logistics resources. This invention evaluates AGV operational status from two dimensions: efficiency and time. Therefore, the AGV status characteristics obtained from the flexible operations digital twin workshop system include the AGV cart... utilization rate Decision point At any moment, the AGV cart The sum of total transmission time and waiting time .

[0055] Equivalent to AGV small car The total transmission time is shorter than that of the AGV trolley. The completion time is a measure of the AGV cart's performance. The core performance indicator of time asset utilization efficiency directly reflects the performance of AGV vehicles. The proportion of effective work. Ideally, it should be maintained at a high level. To avoid resource idleness, but The value cannot be too high, lest the scheduling system lose its flexibility in responding to capacity fluctuations. It is a comprehensive time metric, and the waiting time it includes is a crucial diagnostic indicator, covering congestion waiting caused by path conflicts, idle waiting between task instructions, and waiting time in line at loading or unloading points for robot or operator docking, etc.

[0056] The state space, action space, and reward function of the AGV scheduling agent are defined as follows: 1. Define the state space The state space of the AGV scheduling agent includes the average AGV utilization rate, the standard deviation of the AGV utilization rate, the average AGV completion time, and the standard deviation of the average AGV completion time.

[0057] (1) Average AGV utilization rate As shown in formula (11): (11) (2) Standard deviation of AGV utilization rate As shown in formula (12): (12) (3) Average completion time of AGV As shown in formula (13): (13) (4) Standard deviation of AGV average completion time (VCT) std (t), as shown in formula (14): (14) 2. Define the action space The action space of the AGV scheduling agent is defined as a discrete, finite set. Each basic action in this set directly corresponds to an available AGV in the discrete manufacturing workshop AGV scheduling system integrating digital twins as described in this invention. Specifically, the action space A can be represented as A = {V1, V2, ..., V...} h} where h is the total number of available AGVs in the AGV scheduling system. When the AGV scheduling agent makes a decision, the state characteristics of the AGVs are input into a deep Q-network. The AGV scheduling agent calculates and outputs a corresponding Q-value for each candidate action (i.e., each AGV) in the action space. This Q-value is an estimate of the long-term cumulative reward that can be obtained by selecting a specific AGV. Based on the Q-value, the AGV scheduling agent selects the action with the highest Q-value through, for example, a greedy strategy, that is, selects the AGV number that is most suitable for performing the specific task in the current state, thereby completing accurate task assignment.

[0058] 3. Define the reward function The goal of the AGV scheduling DQN deep neural network model is to improve the efficiency of the entire logistics system by optimizing AGV task allocation. In this invention, the reward function of the AGV scheduling agent aims to achieve two key objectives: first, to shorten the overall completion time of the task flow; and second, to improve the utilization efficiency of AGV resources themselves. To this end, the reward function of the AGV scheduling agent is constructed as a weighted sum of the change in the maximum completion time of AGVs and the change in the average utilization rate of AGVs.

[0059] AGV maximum completion time variation It is defined as the difference between the current decision step and the previous decision step, and the completion time (i.e. the maximum completion time) of the AGV that is the last to complete the task among all AGVs, as shown in formula (15).

[0060] (15) Variation in AGV maximum completion time A decrease in efficiency means an improvement in overall system efficiency, which should be rewarded positively; conversely, an increase in efficiency means a decrease in efficiency, which should be rewarded negatively.

[0061] Change in average AGV utilization It reflects the average change in workload of all AGVs over time, and its calculation formula is as follows (16): (16) Improving AGV utilization helps reduce resource idleness and is an important dimension of scheduling optimization; therefore, it is the change in the average utilization rate of AGVs. An increase in this should bring positive rewards.

[0062] The reward function is composed of the changes in these two indicators, weighted according to their respective importance. and ,like and All values ​​are set to a linear combination of 0.5, and the final instantaneous reward value is stabilized within the range of [-1, 1] by a pruning function to ensure the stability of the training process. Its mathematical definition is shown in formula (17).

[0063] (17) Finally, the AGV scheduling agent consists of one input layer, three hidden layers, and one output layer; the number of nodes in the input and output layers is the same as the number of nodes in the state and action spaces, respectively; the three hidden layers have decreasing numbers of neurons; all hidden layers use the ReLU activation function; the input and output layers do not use activation functions.

[0064] In the discrete manufacturing AGV scheduling system integrating digital twins described in this invention, the flexible operations digital twin workshop system initiates AGV scheduling decisions based on decision points, and deduces and issues these decisions. Decision points are defined as the occurrence of key events in the production system that trigger scheduling requirements, specifically including the arrival of new tasks in the flexible operations digital twin workshop system or the completion of a task. Both of these events change the workshop's state, thus generating new decision requirements. Whenever a decision point t arrives, the flexible operations digital twin workshop system immediately initiates a new data acquisition and processing flow. First, it collects the latest and most accurate multi-dimensional feature parameters in real time from the synchronously running digital twin workshop model. These parameters cover task characteristics, equipment characteristics, and AGV characteristics. Subsequently, this raw data is preprocessed and standardized, and constructed into formats that meet network input requirements, serving as real-time state inputs for the task scheduling agent and the AGV scheduling agent. This process ensures that the task scheduling agent and the AGV scheduling agent always make decisions based on a holographic snapshot of the system state that is highly consistent with the physical workshop, providing a reliable data foundation for collaborative reasoning and decision-making between the task scheduling agent and the AGV scheduling agent.

[0065] Then, based on the preprocessed work tasks and equipment feature parameters, a state vector representing the current overall state of the workshop is constructed. The state vector is used as the input of the task scheduling agent to calculate the Q value of all possible actions in the current state, thereby generating a specific task scheduling strategy.

[0066] Based on the preprocessed AGV feature parameters, a state vector representing the overall state of the AGVs in the current workshop is constructed. The AGV state vector is then used as the input to the AGV scheduling agent to calculate the Q-value of all possible actions in the current state, thereby generating a specific AGV allocation strategy.

[0067] The discrete manufacturing AGV scheduling system integrating digital twins described in this invention simulates and extrapolates the decision-making of the task scheduling agent and the AGV scheduling agent in a flexible operation digital twin workshop system, including the determination of task priorities and execution sequences, and AGV allocation schemes. Until the next decision point t (such as the arrival of a new task or the completion of an existing task) occurs, the latest state of the workshop is captured and updated again. The updated state serves as the benchmark for the next round of input, and is simultaneously fed back to the task scheduling agent and the AGV scheduling agent to drive them to make subsequent collaborative decisions, forming a complete closed-loop optimization process of "state perception-decision-simulation-feedback".

[0068] To achieve continuous optimization and precise execution of the scheduling strategy, the task scheduling agent and the AGV scheduling agent initiate online fine-tuning based on the simulation results of the scheduling decisions by the flexible operations digital twin workshop system. Specifically, the flexible operations digital twin workshop system employs time compression technology to simulate the entire process of AGV material handling and equipment processing within a future production cycle at a speed 5-10 times faster than the real-time clock, based on the scheduling decisions of the task scheduling agent and the AGV scheduling agent. During the simulation, key performance indicators such as task completion rate, task delay rate, and AGV equipment utilization rate are tracked and recorded in real time. Potential path conflicts, resource competition, and other abnormal states are also monitored. Key performance indicators are automatically compared and analyzed against preset performance target thresholds. If key performance indicators fail to meet expected standards (e.g., task delay rate exceeds 15% or equipment utilization rate is below 70%), the online fine-tuning process is immediately initiated.

[0069] The online fine-tuning adopts a phased optimization strategy.

[0070] First, the complete interaction trajectory (including state sequence, action sequence, immediate reward, and final performance metric) generated by the current scheduling decision deduction is stored as an independent training sample in the priority experience replay buffer, where the priority of the sample is dynamically adjusted based on the final performance. Then, a batch of data is extracted from the priority experience replay buffer using a combination of random sampling and importance sampling. The loss function of the current deep Q-network relative to the target deep Q-network is calculated, and the Huber loss function is used to enhance training stability. Finally, gradient descent updates are performed using the Adaptive Moment Estimator (Adam), while setting a gradient pruning threshold to prevent gradient explosion, until the current deep Q-network converges. The target deep Q-network updates its network parameters synchronously using a soft update method with a small time constant, such as τ=0.005. This fine-tuning mechanism enables the job scheduling agent and the AGV scheduling agent to obtain high-quality training samples from the virtual environment, achieving rapid parameter convergence and continuous policy optimization.

[0071] Once the scheduling decision is verified by the flexible operation digital twin workshop system and the optimal scheduling strategy achieves the expected goals, it is dynamically issued to the execution system of the physical workshop through a standardized industrial communication interface (using the OPC UA protocol). To ensure accurate execution of instructions, the abstract scheduling strategy is transformed into a specific sequence of equipment control commands through instruction parsing, including the AGV's pathpoint coordinates, loading and unloading instructions, and machine tool processing program call instructions. These instructions are transmitted in real time to the PLC controller and AGV navigation system in the field via industrial Ethernet, achieving a precise mapping from virtual instructions to physical control.

[0072] The discrete manufacturing workshop AGV scheduling system based on the aforementioned system and the discrete manufacturing workshop AGV scheduling method based on the fusion of digital twins of the present invention include the following steps: S1 constructs a flexible digital twin workshop system based on the layout, equipment parameters, and production process of the physical workshop, endowing all equipment with production control logic and precise geometric attributes.

[0073] S2, based on a deep Q-network solution model, defines a state space with seven state features and an action space with six features that determine the priority and execution sequence rules of the tasks. It constructs a task scheduling agent with the task delay rate, the estimated task delay rate, and the average equipment utilization rate as the core reward functions.

[0074] S3, based on the deep Q-network solution model, defines a state space with four state features, which includes the action space of all available AGVs. The reward function is the weighted sum of the changes in the maximum completion time of AGVs and the changes in the average utilization rate of AGVs, and an AGV scheduling agent is constructed.

[0075] S4 configures the training parameters for the job scheduling agent and the AGV scheduling agent, including the number of training cycles, batch size, discount factor, target network update parameters, exploration rate decay strategy, experience replay buffer size, learning rate, optimizer parameters, and network structure parameters.

[0076] The key hyperparameters of the job scheduling agent are set as follows: The model undergoes L=10 training epochs, and the capacity of the replay buffer is set to N=1000 to store the agent's interaction experience. During gradient descent, the batch size randomly drawn from the buffer is 32. The exploration rate ε in the action selection strategy adopts a linear decay mechanism, gradually decreasing from 0.5 to 0.1 to balance exploration and utilization during training. The reward discount factor γ is set to 0.9 to measure the importance of future rewards. Furthermore, in the soft update strategy of the target network, the parameter τ is set to 0.01 to control the update magnitude of the target network parameters and ensure the stability of the training process. During the training phase, the action of each decision point and each rescheduling point of the job scheduling agent is randomly selected with probability ε. However, when applying the trained DQN algorithm, actions with higher Q values ​​should be prioritized. Conversely, if actions with the highest Q values ​​are selected indiscriminately, the generated scheduling scheme may fall into local optima. After DQN training is complete, a "softmax" strategy is used to select actions. In this strategy, action a is taken at each rescheduling time step (decision point t). i probability Calculated using formula (18): (18) Here, µ is a hyperparameter that controls the softmax policy entropy, which determines the degree of confidence in the estimated values ​​of the trained agent across all execution states.

[0077] The key hyperparameters of the AGV scheduling agent are set as follows: The model performs L=100 training epochs, and the capacity of the experience replay buffer is set to N=10000 to ensure sufficient storage of diverse interaction experiences. During gradient descent, the batch size randomly drawn from the buffer is 32. The exploration rate ε in the action selection strategy adopts a linear decay mechanism, gradually decreasing from 0.9 to 0.1 to encourage full exploration in the early stages of training and shift towards effective utilization later. The reward discount factor γ is set to 0.95, giving higher importance to future rewards to accommodate the long-term impact of scheduling decisions. The learning rate α is set to 0.0005 to ensure stable updates of network weights. Furthermore, in the soft update strategy for the target depth Q-network, the parameter τ is set to 0.005 to achieve smooth and slow updates of the target depth Q-network, further improving training stability. The optimizer uses the Adam algorithm, with its momentum parameter... and The values ​​are set to 0.9 and 0.999 respectively. These parameter designs, in conjunction with the aforementioned deep network architecture (128-64-32 hidden layers), ensure that the AGV scheduling agent can effectively learn high-quality allocation strategies from complex state features.

[0078] S5 defines decision points and extracts multi-dimensional feature parameters in real time from the flexible operation digital twin workshop system, including task features, equipment status features, and AGV status features.

[0079] S6, start simulation training; the job task scheduling agent receives job task characteristics and equipment status characteristics, and outputs job task priority and execution sequence; the AGV scheduling agent receives AGV status characteristics and the output of the job task scheduling agent to obtain the AGV allocation scheme.

[0080] S7 accelerates the simulation of task priorities and execution sequences, as well as AGV allocation schemes in the flexible operation digital twin workshop system, and monitors key performance indicators in real time, including task delay rate and equipment utilization rate. S8 determines whether key performance indicators meet expectations; if not, online fine-tuning is initiated; the simulation trajectory is stored in the experience playback buffer, and the network parameters of the job scheduling agent and the AGV scheduling agent are updated through gradient descent.

[0081] S9, if the expected results are achieved, the task priority and execution sequence, and the AGV allocation scheme are parsed into control instructions for specific equipment, including AGV path coordinates, operating parameters, and machine tool processing programs; these instructions are then sent to the physical workshop execution system through a standardized industrial communication interface.

Claims

1. A discrete manufacturing workshop AGV scheduling system integrating digital twins, characterized in that: This includes a flexible digital twin workshop system, a task scheduling intelligent agent, and an AGV scheduling intelligent agent; The flexible operation digital twin workshop system integrates the layout, equipment parameters and production process of the physical workshop, assigns production control logic to all equipment and simulates the production status of the physical workshop in real time, initiates AGV scheduling decisions based on decision points, and performs and issues scheduling decisions. The task scheduling agent, based on the task characteristics and equipment status characteristics provided by the flexible operation digital twin workshop system, adopts a solution model based on deep Q-networks and outputs a task scheduling strategy, including the priority and execution sequence of tasks, through state space, action space and reward function. The AGV scheduling agent, based on the AGV state characteristics provided by the flexible operation digital twin workshop system, outputs a task scheduling strategy. It adopts a solution model based on a deep Q-network and outputs an AGV allocation strategy through the state space, action space, and reward function.

2. The discrete manufacturing workshop AGV scheduling system integrating digital twins as described in claim 1, characterized in that: The task scheduling agent and AGV scheduling agent initiate online fine-tuning based on the simulation results of the scheduling decisions from the flexible operation digital twin workshop system. The simulation results include task delay rates exceeding thresholds and equipment utilization rates falling below thresholds. The online fine-tuning adopts a phased optimization strategy, placing the simulation trajectory of the current scheduling decision as an independent sample into an experience replay buffer. Sample data is extracted in batches from the experience replay buffer using a combination of random sampling and importance sampling. The loss function of the current depth Q-network relative to the target depth Q-network is calculated, and gradient descent updates are performed using an adaptive moment estimation optimizer. At the same time, a gradient pruning threshold is set to prevent gradient explosion until the current depth Q-network converges. The target depth Q-network updates its network parameters with a fixed time constant.

3. The discrete manufacturing workshop AGV scheduling system integrating digital twins according to claim 1, characterized in that: The task scheduling agent comprises an input layer, an output layer, and five hidden layers. The number of nodes in the input and output layers is the same as the number of nodes in the state and action spaces, respectively. Each hidden layer contains 30 nodes. The input and hidden layers use the Tansig activation function, while the output layer uses the Purelin activation function. The state space of the task scheduling agent includes average equipment utilization, the standard deviation of average equipment utilization, average operation completion rate, average task completion rate, the standard deviation of average task completion rate, estimated task delay rate, and actual task delay rate. The action space of the task scheduling agent includes six rules for determining task priorities and execution sequences. The reward function of the task scheduling agent consists of the actual task delay rate, the estimated task delay rate, and the average equipment utilization.

4. The discrete manufacturing workshop AGV scheduling system integrating digital twins according to claim 1, characterized in that: The AGV scheduling agent comprises an input layer, three hidden layers, and an output layer; the number of nodes in the input and output layers is the same as the number of nodes in the state and action spaces, respectively; the three hidden layers have decreasing numbers of neurons; all hidden layers use the ReLU activation function; the input and output layers do not use activation functions; the state space of the AGV scheduling agent includes the average AGV utilization rate, the standard deviation of the AGV utilization rate, the average AGV completion time, and the standard deviation of the average AGV completion time; the action space of the AGV scheduling agent includes all available AGVs in the discrete manufacturing workshop AGV scheduling system integrating digital twins; the reward function of the AGV scheduling agent is the weighted sum of the change in the maximum AGV completion time and the change in the average AGV utilization rate.

5. The discrete manufacturing workshop AGV scheduling system integrating digital twins according to claim 1, characterized in that: The decision points include when a new job task arrives at the flexible digital twin workshop system or when a job task is completed.

6. The discrete manufacturing workshop AGV scheduling system integrating digital twins according to claim 1, characterized in that: The characteristics of the task include decision points. Time, assignment Number of operations completed Decision point Time, assignment completion rate The device status characteristics include decision points. At any time, equipment Completion time of the last task operation Decision point At any time, equipment average output rate The AGV status characteristics include the AGV trolley. utilization rate Decision point At any moment, the AGV cart The sum of total transmission time and waiting time .

7. The discrete manufacturing workshop AGV scheduling system integrating digital twins according to claim 1, characterized in that: The scheduling decision simulation employs time compression technology, simulating production until the next decision point emerges based on the task scheduling strategy and AGV allocation strategy determined by the task scheduling agent and AGV scheduling agent.

8. The discrete manufacturing workshop AGV scheduling system integrating digital twins according to claim 1, characterized in that: Once the scheduling decision simulation results achieve the expected goals, the scheduling decisions are transmitted to the execution system of the physical workshop through scheduling decision instruction parsing and standardized industrial communication interfaces.

9. A discrete manufacturing workshop AGV scheduling method based on a fusion digital twin of the system described in any one of claims 1-8, characterized in that, Includes the following steps: S1 constructs a flexible digital twin workshop system based on the layout of the physical workshop, equipment parameters, and production process flow, and endows all equipment with production control logic and precise geometric attributes; S2, based on the deep Q-network solution model, defines a state space with seven state features and an action space with six features to determine the priority and execution sequence rules of the tasks. The task scheduling agent is constructed with the task delay rate, the estimated task delay rate and the average equipment utilization rate as the core reward functions. S3, based on the deep Q-network solution model, defines a state space with four state features, which includes the action space of all available AGVs. The reward function is the weighted sum of the change in the maximum completion time of AGV and the change in the average utilization rate of AGV, and an AGV scheduling agent is constructed. S4, configure the training parameters for the job scheduling agent and the AGV scheduling agent respectively, including the number of training cycles, batch size, discount factor, target network update parameters, exploration rate decay strategy, experience replay buffer size, learning rate, optimizer parameters and network structure parameters; S5 defines decision points and extracts multi-dimensional feature parameters in real time from the flexible operation digital twin workshop system, including operation task features, equipment status features and AGV status features; S6, start simulation training; the job task scheduling agent receives job task characteristics and equipment status characteristics and outputs job task scheduling strategy; the AGV scheduling agent receives AGV status characteristics and the output of the job task scheduling agent to obtain AGV allocation strategy. S7, the flexible operation digital twin workshop system accelerates the simulation based on the task scheduling strategy and AGV allocation strategy, and monitors key performance indicators in real time, including task delay rate and equipment utilization rate. S8 determines whether key performance indicators meet expectations; If the expected results are not achieved, online fine-tuning is initiated; the simulated trajectory is stored in the experience replay buffer, and the network parameters of the job scheduling agent and the AGV scheduling agent are updated through gradient descent. S9, if the expected results are achieved, the task scheduling strategy and AGV allocation strategy are parsed into control instructions for specific equipment and sent to the physical workshop execution system through a standardized industrial communication interface.

10. The discrete manufacturing workshop AGV scheduling method integrating digital twins according to claim 9, characterized in that: The specific control instructions for the equipment include AGV path coordinates, operating parameters, and machine tool processing programs.

Images