A Flexible Workshop Production Resource Scheduling Method Based on Multi-Strategy Deep Reinforcement Learning

Abstract

This invention discloses a flexible workshop production resource scheduling method based on multi-strategy deep reinforcement learning, relating to the field of intelligent manufacturing. The method constructs three strategy networks (jobs, machines, and AGVs) and three value networks. At each scheduling moment, the job selection strategy network selects scheduling rules based on the job status and maps them to specific operations; the AGV selection strategy network selects scheduling rules based on the AGV status and maps them to the first AGV for transport operations; the machine selection strategy network maps the machine status to the first machine for processing operations. Through multi-strategy collaboration, the operation is determined to be transported by AGV to the machine for processing. The start and end times of the operation are calculated, and the status, actions, and rewards are stored in the corresponding experience pools. The value networks sample data from the experience pools, calculate losses, and update network parameters. After the experience pools are cleared, the next iteration begins. This method can dynamically respond to changes in production demand and optimize resource allocation and scheduling decisions in real time.

Description

Technical Field

[0001] The invention belongs to the field of intelligent manufacturing, specifically involving a flexible workshop production resource scheduling method based on multi-strategy deep reinforcement learning. Background Technology

[0002] Multi-variety, small-batch production has become an important model in modern manufacturing. Against this backdrop, manufacturing enterprises are increasingly demanding higher product quality and more refined manufacturing processes to meet consumer demand for high-quality and personalized products. With the continuous improvement of the intelligence level of manufacturing workshops, Automated Guided Vehicles (AGVs) play a crucial role in the production process. They are responsible for picking up and transporting raw materials and semi-finished products between machines, forming a complete production resource system in the workshop together with various other machines.

[0003] In such a production scenario, the coordinated scheduling of production resources is particularly important. Coordinated scheduling of production resources aims to effectively integrate and rationally allocate various resources, including machines and AGVs, to ensure the efficient and smooth operation of the production process. However, dynamic events occur frequently in modern customized manufacturing, such as random job arrivals, equipment failures, order cancellations or modifications. These dynamic disturbances may cause the original static scheduling scheme to deviate from expectations during execution, significantly reducing production stability and efficiency. Therefore, when dynamic events occur, there is an urgent need for a comprehensive and coordinated replanning and scheduling of production resources, including machines and AGVs. It is necessary to comprehensively consider factors such as the status of production resources and task priorities to ensure that the scheduling system can quickly respond to and adapt to various emergencies, thereby maintaining the efficient and stable operation of the production resource coordination system and ensuring the successful completion of production tasks.

[0004] To address these challenges, researchers have proposed metaheuristic algorithms (such as genetic algorithms and particle swarm optimization), which can provide near-optimal solutions within a finite time. However, these algorithms still face the challenge of excessive computation time when dealing with large-scale problems, and require extensive iterative computation when the problem size changes. In contrast, Priority Dispatching Rules (PDR) make scheduling decisions quickly through pre-defined rules (such as first-come, first-served, shortest processing time, shortest transportation time, etc.), effectively responding to real-time changes in dynamic events. However, a single PDR can only generate feasible solutions and cannot guarantee global optimality, which can easily lead to production delays, resource idleness, or over-utilization, thereby reducing production efficiency, increasing production costs, and affecting on-time order delivery rates.

[0005] Deep Reinforcement Learning (DRL), a cutting-edge technology in artificial intelligence, has demonstrated immense potential in handling complex decision-making problems. Through continuous interaction between the agent and its environment, DRL can autonomously learn and optimize decision-making strategies to maximize long-term goals. Multi-policy deep reinforcement learning is an advanced optimization method based on the DRL framework, which integrates multiple policies to address complex decision-making problems. In traditional reinforcement learning, agents typically rely on a single policy for decision-making, while multi-policy methods train multiple policies in parallel, making them more adaptable and flexible in different situations. Specifically, multi-policy DRL introduces a policy collaboration mechanism, enabling it to flexibly select the optimal policy or balance multiple policies when facing dynamically changing environments, thus addressing multi-dimensional constraints and uncertainties. Applying multi-policy deep reinforcement learning to the collaborative scheduling of production resources in flexible workshops can not only effectively handle the complex and ever-changing resource allocation and task scheduling problems within the workshop but also optimize the synergistic effects between different types of production resources (such as machines and AGVs). By comprehensively considering multiple production factors such as processing time, order priority, and delivery deadline, multi-strategy deep reinforcement learning can adjust scheduling strategies in real time under the dynamic production demands of flexible workshops, so as to achieve optimal resource allocation and scheduling decisions in the production process. Summary of the Invention

[0006] To overcome the above shortcomings, this invention provides a flexible shop floor production resource scheduling method based on multi-policy deep reinforcement learning, enabling collaborative scheduling of machines and AGVs. This method is based on a flexible shop floor job scheduling model with random job arrivals, utilizing Markov decision processes to simulate the decision-making and collaborative relationships between multiple policies. Furthermore, a reward-sharing mechanism is designed, and a precise reward function is constructed to enhance the synergistic effect between policies, helping to address the issues of solution speed and solution quality under dynamic events.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A flexible shop floor production resource scheduling method based on multi-strategy deep reinforcement learning is proposed. This method employs a multi-strategy cooperation mechanism to construct a decision model including a job selection module, an AGV selection module, and a machine selection module. It schedules flexible shop floor production resources that are randomly reached by jobs. The specific steps are as follows:

[0009] S1. Establish a job selection module, machine selection module, and AGV selection module based on Markov decision-making. The job selection module, machine selection module, and AGV selection module all use optimization algorithms including policy networks and value networks to train the action selection agent. The parameters of the policy networks and value networks of the job selection module, machine selection module, and AGV selection module are independent of each other. Design a composite scheduling rule for job selection, machine selection, and AGV selection.

[0010] S2. Initialize the network parameters of the agents in the job selection module, machine selection module, and AGV selection module, and train them;

[0011] S3. During training, if there is a new job, the new job is added to the set of incomplete jobs and the status of the job, machine and AGV is calculated. Otherwise, the status is calculated directly.

[0012] S4. After the state is determined, multi-strategy collaborative scheduling of job selection, machine selection, and AGV selection is performed at each scheduling time. The outputs of the job selection strategy network, machine selection strategy network, and AGV selection strategy network are sent to... The function obtains the probability distribution of various job selection rules, randomly samples a probability value from the probability distribution, obtains its corresponding index, and selects the corresponding rule from the composite scheduling rules in step S1 based on the index, mapping the job status to the specific operation. Mapping AGV status to transportation operations The AGV is mapped to the processing operation based on the machine status. The machine;

[0013] S5. Determine the operation through multi-strategy collaboration in step S4. Depend on Transport to machine After processing, it enters the next state and performs calculations. The start and end times are calculated based on the average delay time, and the agent's reward is stored in the corresponding experience pool along with the status, actions, and rewards of job selection, machine selection, and AGV selection.

[0014] S6. Repeat steps S3, S4, and S5 until the newly arrived jobs and existing jobs in the workshop have been processed in a single scheduling example. Then, sample single-round data from the experience pool, calculate the loss, update the network parameters, and clear the experience pool to enter the next iteration.

[0015] Further optimization involves introducing residual blocks into both the policy network and the value network. Each residual block comprises two fully connected layers, uses the ReLU activation function, and has skip connections. Multiple residual blocks are stacked in the network, and the output of the last residual block is passed to the fully connected layer of the network for output. The output of the policy network is:

[0016] ,

[0017] in, This represents the output of the last fully connected layer in the policy network. , This represents the weight matrix and bias terms of the fully connected layer. This is the output of the last residual block in the policy network;

[0018] In the value network, the last residual block is passed to the fully connected layer of the network to compute the state value. :

[0019]

[0020] in, This is the output of the last residual block in the value network. , This represents the weight matrix and bias terms of the fully connected layer of the value network.

[0021] Further optimization involves incorporating colored noise into the computations of the job selection strategy network, machine selection strategy network, and AGV selection strategy network. This colored noise is a noise term corresponding to the output dimension of the strategy network. This noise term is added to the original output of the strategy network to form a noisy output. In the strategy network, the colored noise is... Add to :

[0022] ,

[0023] pass Calculate the probability distribution of actions

[0024]

[0025] in, This represents the number of actions.

[0026] To further optimize this, the job selection strategy network, machine selection strategy network, and AGV selection strategy network are updated using a shared instant reward.

[0027] Further optimization is made to include the action space of the composite scheduling rules in step S1, which includes job scheduling rules, machine scheduling rules, and AGV scheduling rules.

[0028] Further optimization is needed; the job scheduling rule is as follows:

[0029] (1) If If the task is empty, select the task with the shortest average relaxation time; otherwise, select the task with the longest overdue time and highest priority for processing.

[0030] (2) If If empty, select the job with the smaller relaxation time critical ratio and higher priority; otherwise, select the job with the longest overdue time and higher priority for processing.

[0031] (3) If If empty, select the job with low completion rate and high priority; otherwise, select the job with the longest overdue time and high priority for processing.

[0032] (4) To avoid getting trapped in local optima, randomly select one job from the set of unfinished jobs for processing;

[0033] in For rescheduling points A collection of overdue assignments.

[0034] Further optimization is needed; the machine scheduling rule is as follows:

[0035] (1) Select the earliest available machine;

[0036] (2) Select the machine with the shortest processing time;

[0037] (3) Select the machine with the lowest utilization rate;

[0038] (4) Select the machine with the lowest load;

[0039] (5) Randomly select a machine that can process the material.

[0040] Further optimization is needed; the AGV scheduling rule is as follows:

[0041] (1) Select the earliest available AGV;

[0042] (2) Select the AGV with the shortest transportation time;

[0043] (3) Select the AGV with the lowest utilization rate;

[0044] (4) Select the AGV with the lowest load;

[0045] (5) Randomly select an available AGV.

[0046] Further optimization involves the job selection strategy network in step S4 mapping the job scheduling rules to specific operations. The AGV selection strategy network maps AGV scheduling rules to transportation operations. The Taiwan AGV The machine selection strategy network maps machine states to processing operations. The Taiwanese machine .

[0047] Further optimization yields the following method for generating colored noise sequences: First, based on the length of the input noise... Calculate the frequency components The set, based on the color parameters of the noise Calculate the scaling factor for each frequency. The spectral density is obtained by combining scaling factors. And calculate the standard deviation. Used for normalization, then based on standard deviation and scaling factor Random numbers conforming to a normal distribution are generated and used for the real and imaginary parts of the spectrum, respectively. The real and imaginary parts are combined to generate a complex spectrum. An inverse Fourier transform is applied to the generated complex spectrum to obtain a noise sequence in the time domain. The generated noise sequence is normalized to obtain the final colored noise sequence.

[0048] The beneficial effects of this invention are as follows:

[0049] 1. A multi-strategy collaborative mechanism is adopted, including job selection, machine selection, and AGV selection. The decision-making process is based on the Markov decision model. In each selection module, a proximal policy optimization algorithm with a policy-value system is used to train the action selection agent. These two networks are implemented through a multilayer perceptron. The function of the policy network is to generate the probability distribution of different actions to guide the agent's decision-making. The value network is responsible for estimating the value function of different states and providing support for the training of the policy network. The parameters of the policy network and the value network in each selection module are independent of each other and are not shared.

[0050] 2. Residual blocks are introduced into each policy network and value network to effectively improve network performance and enable gradients to be transmitted more directly between different layers, thereby effectively alleviating the gradient vanishing problem. Colored noise is added to the policy network to enhance the agent's exploration ability and help break through local optima.

[0051] In summary, multi-strategy DRL, by introducing a strategy collaboration mechanism, can flexibly select the optimal strategy or balance multiple strategies when facing dynamically changing environments, in order to cope with multi-dimensional constraints and uncertainties. Applying multi-strategy deep reinforcement learning methods to the collaborative scheduling of production resources in flexible workshops can not only effectively handle the complex and ever-changing resource allocation and task scheduling problems in the workshop, but also optimize the synergistic effects between different types of production resources in the workshop. By comprehensively considering multiple production factors such as processing time, order priority, and delivery deadline, multi-strategy deep reinforcement learning can adjust scheduling strategies in real time under the dynamically changing production demands of flexible workshops, so as to achieve optimal resource allocation and scheduling decisions in the production process. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the operation process of the present invention;

[0053] Figure 2 This is a schematic diagram of the strategy network and value network structure;

[0054] Figure 3 A schematic diagram of a dynamic event-driven collaborative scheduling scenario for production resources;

[0055] Figure 4 This is a schematic diagram of a multi-strategy collaboration framework;

[0056] Figure 5 This is a schematic diagram illustrating the generation process of a colored noise sequence.

[0057] Figure 6 for A schematic diagram of the calculation process;

[0058] Figure 7 This is a schematic diagram of the training process of the MPPPO training algorithm for multi-strategy collaboration. Detailed Implementation

[0059] A flexible workshop production resource scheduling method based on multi-policy deep reinforcement learning, such as Figure 1As shown, this method employs a multi-strategy collaborative mechanism, including job selection, machine selection, and automated guided vehicle selection. The decision-making process is based on a Markov Decision Process (MDP). In each selection module, a proximal policy optimization (PPO) algorithm with a policy-value (Actor-Critic) architecture is used to train the action selection agent. These two networks are implemented using a multilayer perceptron (MLP). The Actor network generates probability distributions for different actions to guide the agent's decision-making; the Critic network estimates the value function for different states, supporting the training of the policy network. The parameters of the policy network and value network in each selection module are independent and not shared.

[0060] Policy network: The network accepts the current state. As input, each residual block consists of two fully connected layers using the ReLU activation function and has skip connections. The following is the formula representation of the residual block:

[0061]

[0062]

[0063] in, This is the output of the first fully connected layer in the residual block. This is the output of the second fully connected layer in the residual block. The input to the residual block is the state feature, specifically the first layer of the first residual block. , These are the weight matrix and bias terms of the first fully connected layer. , These are the weight matrix and bias terms of the second fully connected layer. The formula for calculating skip connections is as follows:

[0064]

[0065] Output As the output of the residual block, it is connected to the next layer. Multiple residual blocks can be stacked, assuming there are... The residual block, the first The output of each residual block is:

[0066]

[0067] in and The first The weight matrix and bias terms of the first fully connected layer in each residual block. and The first The weight matrix and bias terms of the second fully connected layer in each residual block.

[0068] Pass the output of the residual block to the fully connected layer of the policy network:

[0069]

[0070] in, This represents the output of the last fully connected layer in the policy network. , This represents the weight matrix and bias terms of the fully connected layer. (The text also mentions colored noise.) Add to :

[0071]

[0072] Finally, through Calculate the probability distribution of the action:

[0073]

[0074] in, For the number of actions, For the noisy output at the rescheduling time point t, This is the noisy output corresponding to the i-th action.

[0075] Value Network: Similar to policy networks, value networks also accept states. As input, the residual blocks of the value network are consistent with the policy network and stacked. There are several residual blocks, and the formula is as follows:

[0076]

[0077] The calculation formula is consistent with the policy network, and each residual block contains two fully connected layers and a skip connection. The output of the residual block is passed to the fully connected layer to calculate the state value.

[0078]

[0079] in, This is the output of the last residual block in the value network. , This represents the weight matrix and bias terms of the fully connected layer.

[0080] The advantage function is determined by the value network and actual reward. The value of the advantage function is equal to the actual reward minus the value network's estimate of the current state's value. If the advantage function is positive, it means that the action is better than average, and the policy network should increase the probability of executing the action; otherwise, it should decrease the probability.

[0081] The flexible workshop production resource scheduling method based on multi-strategy deep reinforcement learning has the following specific steps:

[0082] S1. Construct a Markov decision model based on the agent model of job selection, machine selection and AGV selection, and design the action space of composite scheduling rules. The action space of composite scheduling rules in step S1 includes job scheduling rules, machine scheduling rules and AGV scheduling rules.

[0083] S2. Initialize the network parameters of the job selection, machine selection, and AGV selection agents, and input them into the above model for training;

[0084] S3. Based on the training in step S2, determine whether the training has ended:

[0085] If the training is not finished, check if there is a new task. If there is a new task, add it to the set of incomplete tasks and calculate the status of the task, machine and AGV. Otherwise, calculate the status directly.

[0086] If the training is completed, then determine whether to terminate the task.

[0087] S4. After the state is determined, multi-strategy collaborative scheduling of job selection, machine selection, and AGV selection is performed at each scheduling time. Colored noise is added to the calculations of the job selection strategy network, machine selection strategy network, and AGV selection strategy network. The colored noise is the noise term corresponding to the output dimension of the strategy network. This noise term is added to the original output of the strategy network to form a noisy output, which is then fed into... In the function, the probability distribution of multiple job scheduling rules is obtained. A probability value is randomly sampled from the probability distribution and its corresponding index is obtained. Based on the index, the corresponding rule is selected from the composite scheduling rules in step S1 and mapped to the specific operation, transportation operation and processing operation of the target task.

[0088] The method for generating the colored noise sequence is as follows: Figure 5 As shown, the color parameters Set to 0.5, noise length Set the output length of the policy network; lines 1-2 first determine the length based on the input. Calculate the frequency components The set; lines 3-4 are based on the noise Calculate the scaling factor for each frequency. Lines 5-6 combine scaling factors to obtain the spectral density. And calculate the standard deviation. Lines 7-8 generate random numbers conforming to a normal distribution based on the standard deviation and scaling factor, used for the real and imaginary parts of the spectrum, respectively. Lines 9-12 target the 0 frequency and Nyquist frequency, which only exist when H is even. These require special handling to ensure the generated noise has correct symmetry, where H is the AGV's usage time from the start to the rescheduling point t. Line 13 generates a complex spectrum by combining the real and imaginary parts. Line 14 applies an inverse Fourier transform to the generated complex spectrum to obtain a noise sequence in the time domain. Line 15 normalizes the generated noise sequence to obtain the final colored noise sequence.

[0089] S5. Perform the operation through multi-strategy collaboration in step S4. By AGV Transport to machine Processing and calculation operations Start and end times, and the status of job selection, machine selection, and AGV selection. ,action and rewards Store in the corresponding experience pool (like Figure 4 As shown in the figure, where t is the rescheduling time point. , , The states, actions, and rewards of the job, machine, and AGV at the rescheduling time point t are respectively defined. The job selection strategy network, machine selection strategy network, and AGV selection strategy network of this invention are updated using shared real-time rewards.

[0090]

[0091] in, For state The corresponding average delay time, For state The corresponding average delay time, This represents the job status at time t+1. The machine corresponding to time t+1. This refers to the AGV state at time t+1.

[0092] S6. Repeat steps S3, S4, and S5 until the newly arrived jobs and existing jobs in the workshop have been processed in a single scheduling example. Then, sample single-round data from the experience pool, calculate the loss, update the network parameters, and clear the experience pool to enter the next iteration.

[0093] Steps S1, S2, S3, and S4 constitute the Markov Decision Process (MDP) for the three modules of job selection, machine selection, and AGV selection.

[0094] Job Selection Module MDP Process

[0095] Step 1: Construct job scheduling rules and calculate the rescheduling point. (Completion of assignments or arrival of new assignments) Collection of overdue assignments and incomplete assignments set ,in Indicate homework Completed operations Indicate homework Total number of operations, This indicates the average completion time for the machine to finish the previous task. Indicate homework The completion time of the current operation.

[0096] Job scheduling rules :if If empty, select the job with the shortest average relaxation time. Otherwise, select the task with the longest overdue period and highest priority for processing. ;

[0097] Job scheduling rules :if If empty, select jobs with a small relaxation time critical ratio and high priority. Otherwise, select the task with the longest overdue period and highest priority for processing. ;

[0098] Job scheduling rules :if If empty, select jobs with low completion rates and high priority. Otherwise, select the task with the longest overdue period and highest priority for processing. ;

[0099] Job scheduling rules To avoid getting trapped in local optima, in the set of unfinished tasks In the process, a task is randomly selected for processing. ;

[0100] in, Represented as homework The degree of urgency, Indicate homework Estimated processing time for the remaining operations. Indicate homework Estimated transport time for the remaining operations. This represents the overdue time for the i-th assignment.

[0101] In summary, at the rescheduling point time point Choose an action for the assignment .

[0102] Step 2: Calculate each rescheduling point Status characteristics of the job , where: average completion rate of operations Average completion rate of assignments Estimate the proportion of overdue work to the total number of work tasks. The proportion of actual overdue work to the total number of work tasks , Indicate homework completion rate This indicates the total number of assignments. , The calculation is as follows:

[0103]

[0104] in, Indicate homework At the rescheduling point The estimated set of overdue operations, express Number of operations that have exceeded the time limit.

[0105]

[0106] At the rescheduling time point The actual set of overdue tasks.

[0107] This invention will reschedule time points and The differences between states are also included as part of the state representation, and the final input to the policy network of the job selection module is... ,in Indicates the current rescheduling time point state, Indicates the current rescheduling time point The state and the previous scheduling time point The difference in states.

[0108] Step 3: The neural network structure of the job selection strategy network is as follows Figure 2 As shown, based on the input The generated raw output Its dimension is 4, corresponding to 4 job scheduling rules. Algorithm 1 generates a 4-dimensional colored noise term. and the original output Add them together to form a noisy output. Then, the noisy output. Sent to The function yields the probability distributions of four job scheduling rules:

[0109]

[0110] in, This represents the probability distribution of the job scheduling rules;

[0111] Subsequently, a probability value is randomly sampled from the probability distribution, and its corresponding index is obtained. Based on this index, the corresponding rule is selected from the job scheduling rules in step one and mapped to the target job. Specific operations .

[0112] Machine Selection Module (MDP) Process

[0113] Step 1: Build the following machine scheduling rules

[0114] Machine scheduling rules Choose the earliest available machine. ;

[0115] Machine scheduling rules Choose the machine with the shortest processing time. ;

[0116] Machine scheduling rules Choose the machine with the lowest utilization rate. ;

[0117] Machine scheduling rules Choose the machine with the lowest load. ;

[0118] Machine scheduling rules Randomly select a machine capable of processing. ;

[0119] in, Indicates the first The time it takes for the machine to complete the previous processing task. Indicates operation In the machine The value is 1 if processed, otherwise it is 0. Indicates operation On the plane On the processing timer, Indicates the machine at the rescheduling time point The maximum completion time. In summary, at the rescheduling point... Machine selection action ,in This is the set of actions that the machine can choose from.

[0120] Step 2: Calculate each rescheduling time point Machine status characteristics Among them, the average utilization rate of machines Standard deviation of machine utilization The ratio of the machine's maximum workload to its average workload The ratio of the standard deviation of machine workload to the average machine workload , Indicates machine utilization rate Indicates the number of machines. Indicates the first The workload of the machine. This indicates the machine's average workload.

[0121] This invention will reschedule time points and The differences between states are also included as part of the state representation, and the final input to the policy network of the machine selection module is... .

[0122] Step 3: The neural network structure of the machine selection strategy network is as follows Figure 2 As shown, the network input is based on the policy of the machine selection module. The generated raw output is Its dimension is 5, corresponding to 5 machine scheduling rules. Algorithm 1 generates a 5-dimensional colored noise term. and the original output Add them together to form a noisy output. Then, the noisy output. Sent to The function yields the probability distributions of five machine scheduling rules:

[0123]

[0124] Subsequently, a probability value is randomly sampled from the probability distribution, and its corresponding index is obtained. Based on this index, the corresponding rule is selected from the machine scheduling rules in step one and mapped to the machine. .

[0125] AGV Selection Module MDP Process

[0126] Step 1: Construct the following AGV scheduling rules

[0127] AGV scheduling rules Select the earliest available AGV ;

[0128] AGV scheduling rules Choose the AGV with the shortest transportation time. ;

[0129] AGV scheduling rules Choose the AGV with the lowest utilization rate. ;

[0130] AGV scheduling rules Select the AGV with the lowest load. ;

[0131] AGV scheduling rules Randomly select an available AGV ;

[0132] in, Indicates the index of the AGV. Indicates that it can be used for transportation operations A collection of AGVs express Transport the previous task The end time of transportation, Indicates operation The completion time, express Use the If the AGV is used for transportation, the value is 1; otherwise, it is 0. Indicates use Transportation operations The delivery time This indicates the AGV at the rescheduling time point. The maximum transport completion time. In summary, the rescheduling time point... AGV select action ,in This is the set of optional actions for the AGV.

[0133] Step 2: Calculate each rescheduling point AGV status characteristics Among them, the average utilization rate of AGV AGV utilization standard deviation The ratio of AGV's maximum working load to its average working load The ratio of the standard deviation of AGV workload to the average machine workload , Indicates AGV utilization rate Indicates the number of AGVs. Indicates AGV The workload, Indicates AGV The average workload.

[0134] This invention will reschedule time points and The differences between states are also included as part of the state representation, and the final input to the AGV selection module's strategy network is... .

[0135] Step 3: The neural network structure of the AGV selection strategy network is as follows Figure 2 As shown, the strategy network input of the AGV selection module is... Generate the original output as Its dimension is 5, corresponding to 5 AGV scheduling rules. Algorithm 1 generates a 5-dimensional colored noise term. and in the original output Add them together to form a noisy output. Then, the noisy output. Sent to The function yields the probability distributions of five AGV scheduling rules:

[0136]

[0137] Subsequently, a probability value is randomly sampled from the probability distribution, and its corresponding index is obtained. Based on this index, the corresponding rule is selected from the AGV scheduling rules in step one and mapped to the AGV. .

[0138] In step S5, the reward is given after job selection, machine selection, and AGV selection, using an AGV. Transportation operations To the machine After processing, the environment transitions from the current state to the next state, and rewards are provided to the agent to aid its learning. The job selection strategy network, machine selection strategy network, and AGV selection strategy network of this invention are updated using shared, instantaneous rewards.

[0139] ,

[0140] in The calculation process is Algorithm 2 (e.g.) Figure 6 As shown), where This indicates the current rescheduling time.

[0141] Algorithm 3 (e.g.) Figure 7 The image shows the training process of the MPPPO training algorithm for multi-strategy collaboration. Lines 2-13 in the algorithm represent the core scheduling process for a single scheduling instance. During this process, three agents collect experience tuples. After the scheduling phase ends, the job rule selection agent, machine rule selection agent, and AGV rule selection agent respectively utilize... Figure 7 The PPO algorithm for neural network architectures (lines 14-24) independently calculates the error and updates the parameters via gradient descent, where line 20... The value is 0.2. This represents the clipping function, which restricts the probability ratio. Range of variation:

[0142] .

[0143] The dynamic events considered in this invention are new tasks with different levels of urgency. They arrive at the workshop randomly. These tasks require processing on multiple machines. Each assignment Contains a specific sequence of operations Each operation It can be made by a set of available machines Processing, and the processing time of this operation on different machines. different, This indicates the machine index. The arrival information for a new job is unknown before it enters the workshop; its arrival time... Following an exponential distribution, the overdue period is set as follows: The arrival time of the existing operations in the workshop is 0. When the new operation arrives at time... Upon arrival at the workshop, the scheduling system triggers a rescheduling process. Based on the urgency and priority of the tasks, the system re-plans all incomplete tasks and the operations of newly arrived tasks, generating a new production plan. For example... Figure 3 As shown, according to the plan, AGVs transport semi-finished products to the buffer zone of designated machines (such as...). Figure 3 (as shown in (a)). When two consecutive operations of a semi-finished product task require processing on different machines, the AGV is responsible for transporting the semi-finished product task from one machine to the target machine (e.g., ...). Figure 3 (b) As shown. If the AGV receives a second task after completing the first transport task, it can go directly to the next target machine without returning to the warehouse (e.g., Figure 3 (c) As shown. When there are no new transport tasks, the AGV will return to the warehouse to standby (e.g., Figure 3 (as shown in (d)). Once an operation is completed, the relevant machine selects the next operation to be processed from the buffer and continues the production process.

[0144] Multi-strategy DRL, by introducing a strategy collaboration mechanism, can flexibly select the optimal strategy or balance multiple strategies in the face of dynamically changing environments. This addresses multi-dimensional constraints and uncertainties, namely, simultaneously assigning each job to the appropriate machine, sorting jobs on the machine, selecting suitable AGVs, and determining the start time of each operation. Using mean tardiness (MT) as the objective function, it should be minimized, mathematically described as follows:

[0145]

[0146] in, Indicate homework The actual completion time Indicates the number of tasks.

[0147] This invention can not only effectively handle the complex and ever-changing resource allocation and task scheduling problems in the workshop, but also optimize the synergistic effect between different types of production resources in the workshop. By comprehensively considering multiple production factors such as processing time, order priority, and delivery deadline, multi-strategy deep reinforcement learning can adjust the scheduling strategy in real time under the dynamic production demand of the flexible workshop, so as to achieve the optimal resource allocation and scheduling decision in the production process.

[0148] The foregoing has shown and described the main features, usage methods, basic principles, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention based on actual circumstances without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A flexible workshop production resource scheduling method based on multi-strategy deep reinforcement learning, characterized in that, A multi-strategy collaboration mechanism is adopted to construct a decision model that includes a job selection module, an AGV selection module, and a machine selection module. This model is used to schedule flexible workshop production resources where jobs are randomly accessed. The specific steps are as follows: S1. Establish a job selection module, machine selection module, and AGV selection module based on Markov decision-making. The job selection module, machine selection module, and AGV selection module all use optimization algorithms including policy networks and value networks to train the action selection agent. The parameters of the policy networks and value networks of the job selection module, machine selection module, and AGV selection module are independent of each other. Design a composite scheduling rule for job selection, machine selection, and AGV selection. S2. Initialize the network parameters of the agents in the job selection module, machine selection module, and AGV selection module, and train them; S3. During training, if there is a new job, the new job is added to the set of incomplete jobs and the status of the job, machine and AGV is calculated. Otherwise, the status is calculated directly. S4. After the state is determined, multi-strategy collaborative scheduling of job selection, machine selection, and AGV selection is performed at each scheduling time. The outputs of the job selection strategy network, machine selection strategy network, and AGV selection strategy network are sent to... The function obtains the probability distribution of various job selection rules, randomly samples a probability value from the probability distribution, obtains its corresponding index, and selects the corresponding rule from the composite scheduling rules in step S1 based on the index, mapping the job status to the specific operation. Mapping AGV status to transportation operations The AGV is mapped to the processing operation based on the machine status. The machine; S5. Determine the operation through multi-strategy collaboration in step S4. Depend on Transport to machine After processing, it enters the next state and performs calculations. The start and end times are calculated based on the average delay time, and the agent's reward is stored in the corresponding experience pool along with the status, actions, and rewards of job selection, machine selection, and AGV selection. S6. Repeat steps S3, S4, and S5 until the newly arrived jobs and existing jobs in the workshop have been processed in a single scheduling example. Then, sample single-round data from the experience pool, calculate the loss, update the network parameters, and clear the experience pool before entering the next iteration. Both the policy network and the value network incorporate residual blocks. Each residual block consists of two fully connected layers, uses the ReLU activation function, and has skip connections. Multiple residual blocks are stacked in the network, and the output of the last residual block is passed to the fully connected layer of the network for output. The output of the policy network is: ， in, This represents the output of the last fully connected layer in the policy network. , This represents the weight matrix and bias terms of the fully connected layer. This is the output of the last residual block in the policy network; In the value network, the last residual block is passed to the fully connected layer of the network to compute the state value. : in, This is the output of the last residual block in the value network. , This represents the weight matrix and bias terms of the fully connected layer of the value network; Colored noise is added to the operation of the job selection strategy network, machine selection strategy network, and AGV selection strategy network. This colored noise is a noise term corresponding to the output dimension of the strategy network. This noise term is added to the original output of the strategy network to form a noisy output. In the strategy network, the colored noise... Add to : ， pass Calculate the probability distribution of actions in, Number of actions; For the noisy output at the rescheduling time point t, This is the noisy output corresponding to the i-th action; The job selection strategy network, machine selection strategy network, and AGV selection strategy network are updated using shared instant rewards; The action space of the composite scheduling rule in step S1 includes job scheduling rules, machine scheduling rules, and AGV scheduling rules. The job scheduling rule is as follows: (1) If If the task is empty, select the task with the shortest average relaxation time; otherwise, select the task with the longest overdue time and highest priority for processing. (2) If If empty, select the job with the smaller relaxation time critical ratio and higher priority; otherwise, select the job with the longest overdue time and higher priority for processing. (3) If If empty, select the job with low completion rate and high priority; otherwise, select the job with the longest overdue time and high priority for processing. (4) To avoid getting trapped in local optima, randomly select one job from the set of unfinished jobs for processing; in For rescheduling points Overdue assignment set; The method for generating colored noise sequences is as follows: First, based on the length of the input noise... Calculate the frequency components The set, based on the color parameters of the noise Calculate the scaling factor for each frequency. The spectral density is obtained by combining scaling factors. And calculate the standard deviation. Used for normalization, then based on standard deviation and scaling factor Generate random numbers that conform to a normal distribution and use them for the real and imaginary parts of the spectrum, respectively. Combine the real and imaginary parts to generate a complex spectrum. Apply an inverse Fourier transform to the generated complex spectrum to obtain a noise sequence in the time domain. Normalize the generated noise sequence to obtain the final colored noise sequence.

2. The flexible workshop production resource scheduling method based on multi-policy deep reinforcement learning as described in claim 1, characterized in that, The machine scheduling rule is as follows: (1) Select the earliest available machine; (2) Select the machine with the shortest processing time; (3) Select the machine with the lowest utilization rate; (4) Select the machine with the lowest load; (5) Randomly select a machine that can process the material.

3. The flexible workshop production resource scheduling method based on multi-policy deep reinforcement learning as described in claim 1, characterized in that, The AGV scheduling rules are as follows: (1) Select the earliest available AGV; (2) Select the AGV with the shortest transportation time; (3) Select the AGV with the lowest utilization rate; (4) Select the AGV with the lowest load; (5) Randomly select an available AGV.

4. The flexible workshop production resource scheduling method based on multi-policy deep reinforcement learning as described in claim 1, characterized in that, In step S4, the job selection strategy network selects scheduling rules based on job status and maps them to specific operations. The AGV selection strategy network selects scheduling rules based on the AGV status and maps them to transportation operations. The Taiwan AGV The machine selection strategy network maps machine states to processing operations. The Taiwanese machine .

Images