Flexible job shop scheduling optimization method and system for limited multi-type transportation resource constraints

Abstract

The application discloses a flexible job shop scheduling optimization method and system for limited multi-type transportation resource constraints, and belongs to the technical field of deep reinforcement learning and production scheduling. The method comprises the following steps: acquiring constraint items and optimization objectives of the flexible job shop scheduling problem with limited multi-type transportation resource constraints, and constructing a flexible job shop scheduling model; modeling the decision-making process of the job shop scheduling model as a multi-agent MDP, defining three types of agents, i.e., process agents, machine agents and AGV agents, based on a multi-agent system of the PPO-CTDE architecture, performing network training and optimization in the interaction stage and the update stage, and solving the flexible job shop scheduling problem with limited multi-type transportation resource constraints. Through the distributed cooperation mechanism of the multi-agent and the efficient learning ability of the PPO algorithm in the continuous decision-making space, the application realizes dynamic adaptation of transportation resources and processing resources and global collaborative optimization, improves the overall scheduling efficiency and reduces the comprehensive cost.

Description

Technical Field

[0001] The present invention relates to the technical fields of deep reinforcement learning and production scheduling, and specifically provides a flexible job shop scheduling optimization method and system for limited multi-type transportation resource constraints. Background Technique

[0002] With the development background of Industry 4.0 and intelligent manufacturing, the traditional fixed automated production mode has been difficult to meet the market's demands for high flexibility, high efficiency, and high customization. The flexible manufacturing system has emerged as the core production mode in fields such as automobiles, aviation, and high-end equipment. As a key link connecting production plans and shop floor execution, production scheduling technology directly determines the operating efficiency and cost control of the flexible manufacturing system by dynamically optimizing resource allocation and process flows.

[0003] However, the efficient coordination between production scheduling and logistics transportation still faces challenges. Especially in the transportation link, the differences in the size, weight, etc. of different workpieces require matching different types of transportation resources (such as multi-type AGVs), which increases the complexity of transportation organization and scheduling. Existing research mostly focuses on single transportation resource or flexible scheduling problems without transportation constraints. Although mathematical programming and heuristic algorithms have achieved certain results, they are difficult to handle the dynamic adaptation and global optimization problems of multi-type transportation resources, and have deficiencies such as limited computational scale, strong parameter dependence, and decision-making lag.

[0004] In recent years, due to its advantages in dealing with dynamic and complex decision-making problems, deep reinforcement learning has gradually been applied to the field of production scheduling. However, the traditional single-agent deep reinforcement learning model still has limitations when dealing with multi-type transportation resource constraints. It is difficult to simultaneously consider multi-dimensional objectives such as operation sequencing, equipment allocation, and transportation coordination, and is prone to falling into local optima. It is difficult to adapt to the complexity and strong coupling problems in large-scale, high-dynamic production environments and transportation-production integrated scheduling scenarios. Moreover, the single-agent method is difficult to represent the dynamic association between multi-type transportation and processing resources and lacks the ability of global collaborative decision-making, resulting in unbalanced resource matching, insufficient production-transportation coordination, and causing scheduling inefficiency and resource waste. Therefore, there is an urgent need for an intelligent system that can collaboratively optimize multi-type transportation resources and production scheduling to achieve the true global optimization of production and logistics links. Summary of the Invention

[0005] To solve the above problems, the present invention provides a flexible job shop scheduling optimization method and system for limited multi-type transportation resource constraints to solve the complexity and strong coupling problems in transportation-production integrated scheduling scenarios.

[0006] To achieve the above object, the present invention provides the following technical solutions: A flexible job shop scheduling optimization method and system for limited multi-type transportation resource constraints, specifically including the following steps: S1. Obtain the constraint terms and optimization objective of the flexible job shop scheduling problem with limited multi-type transportation resources, and construct a flexible job shop scheduling model; S1.1 The optimization target is set as the maximum completion time, expressed as: (1); In the formula, The maximum completion time for all workpieces. For process In the machine The processing completion time on the surface For the workpiece index, and ; For machine indexing, and .

[0007] S1.2 Obtain the constraint terms of the flexible job shop scheduling problem under the constraint of finite multi-type transportation resources, including: (1) Uniqueness constraint of workpiece type, ensuring that each workpiece belongs to only one type; (2) AGV type uniqueness constraint to ensure that each AGV corresponds to only one type; (3) Transportation type matching constraints to ensure that the execution of transportation tasks meets the consistency requirements between the workpiece and the AGV type; (4) Uniqueness constraint for process-AGV allocation, ensuring that any process can only be assigned to a single type of AGV; (5) Machine and AGV process continuity constraints ensure that all processes on any machine or AGV assigned a task can form a continuous, acyclic work sequence. (6) Consistency constraint of machine processing sequence, to ensure that the sequence of any two processes on the same machine should be consistent; (7) AGV transportation task sequence constraints to ensure that any two transportation tasks undertaken by the same AGV should be consistent; (8) Processing time sequence constraints to ensure that the processing completion time of any process is not earlier than its processing start time; (9) Processing-transportation start time constraints to ensure that the processing start time of any process is not earlier than its transportation arrival time; (10) Machine-AGV path conflict avoidance constraint to ensure that two processes processed on the same machine will not have a transportation task; (11) Transportation - Precedence completion time constraints, ensuring that the transportation arrival time of any process is not earlier than the processing completion time of its predecessor process; S2. Model the decision-making process of the workshop scheduling model as a multi-agent Markov decision process (MDP), and define three types of agents: process agent, machine agent, and automated guided vehicle (AGV) agent. The process intelligence agent is used for sorting and prioritizing unprocessed processes; The machine intelligence agent is used to allocate the optimal processing equipment to the target process selected by the process intelligence agent; The automated guided vehicle (AGV) intelligent agent is used for the target process selected by the process intelligent agent to complete the optimal transportation scheduling and collaborative connection of processed materials; S2.1 Defines the environment for MDP; S2.2. Based on the MDP environment, construct the state space of process intelligent agent, machine intelligent agent and AGV intelligent agent; S2.2.1 Define the state characteristics of the process; In this invention It represents the status characteristics of all current processes, specifically including 5 characteristic information items; (1) In Whether all workpieces have been processed at any given time. This is the moment for making a decision.

[0008] (2) In The average completion rate of all processes at any given time.

[0009] (3) In The average completion rate of all workpieces at any given time.

[0010] (4) In The standard deviation of the average completion rate of all workpieces at any given time.

[0011] (5) In The average remaining process rate for all workpieces at any given time.

[0012] S2.2.2 Define the state characteristics of the machine; In this invention, It represents the current state characteristics of all machines, specifically including three characteristic information items; (1) In The earliest available time for all machines at any given moment.

[0013] (2) In The average utilization rate of all machines at any given time.

[0014] (3) In The standard deviation of the average utilization rate of all machines at any given time.

[0015] S2.2.3 Define the state characteristics of the AGV; In this invention, This indicates the current status characteristics of all AGVs, which specifically includes three feature information items; (1) In The earliest available time for all AGVs at any given moment.

[0016] (2) In The average utilization rate of all AGVs at any given time.

[0017] (3) In The standard deviation of the average utilization rate of all AGVs at any given time.

[0018] S2.3. Based on the MDP environment, construct the motion space of process intelligent agents, machine intelligent agents, and AGV intelligent agents; S2.4. Based on the optimization objective, define three types of agent rewards; In the multi-agent cooperative scheduling system of this invention, due to the adoption of a centralized training and distributed execution CTDE architecture, the common optimization objective of all agents is to minimize the maximum completion time of the workpiece. Therefore, the time interval is defined. The single-step reward function is: .

[0019] Cumulative rewards for the entire round The sum of the single-step rewards at each time step is: .

[0020] In the formula, The start time of scheduling Indicates the current decision-making moment state, This represents the total number of decision-making steps. Therefore, to maximize cumulative rewards, the agent needs to minimize the scheduling termination time. Total completion time This is completely consistent with the core optimization objective of the system of this invention.

[0021] S2.5 Define the state transition process of the MDP; when the three types of agents output a joint action Then, based on the flexible workshop scheduling model, the environment updates the current state of the workshop and calculates and generates the state for the next moment; S3. Based on the proximal strategy optimization of the PPO algorithm and the centralized training and distributed execution CTDE architecture, construct a multi-agent system based on the PPO-CTDE architecture; The multi-agent system includes a policy network of process agents, machine agents, and AGV agents, as well as a centralized value network. S4. A multi-agent system based on the PPO-CTDE architecture performs network training and optimization in the interaction and update phases to solve the flexible job shop scheduling problem with constraints of limited multi-type transportation resources, and completes the flexible job shop scheduling optimization method and system for constraints of limited multi-type transportation resources. Specifically, the following steps are included: S4.1 Interaction Phase: During the interaction between the agent and the environment, the parameters of its policy network remain fixed. The process agent, machine agent, and AGV agent execute distributed decision-making in parallel based on their respective local observations, generating a joint action vector. And it acts on the environment; the environment performs state transitions according to the joint action and the physical constraints of the scheduling model, and feeds back immediate rewards and the next state, generating and storing interaction experience data to the experience storage pool; S4.2 Update Phase: The global state is evaluated based on the experience storage pool. After eliminating the deviation between the evaluation network prediction value and the target state value by calculating and minimizing the mean square error loss function, the advantage function is calculated and the near-end strategy is used to optimize the policy loss objective function of PPO. The policy network parameters of process agent, machine agent and AGV agent are updated synchronously. Minimize the mean squared error loss function The expression is: In the formula, To evaluate the network predictions, Value of the target state; Advantage function The expression is as follows: In the formula, For a moment The single-step reward function, This is the discount value coefficient; The expression for the policy loss objective function is: In the formula, For truncation terms, The probability ratio between the old and new strategies. For the safe zone, This is the entropy regularization term.

[0022] Beneficial effects of this invention: This invention constructs a multi-agent cooperative scheduling system based on Process Optimization (PPO) to address the problem of flexible job shop scheduling (FJSP-LTTRs) with limited and diverse transportation resources. It innovatively proposes a real-time response mechanism using a centralized training and distributed execution CTDE architecture, effectively breaking down the strong coupling between production and logistics, as well as the scheduling complexity arising from the workshop environment. The system relies on the distributed collaborative logic of process agents, machine agents, and AGV agents. Through precise perception and modeling of workpiece processing progress, equipment load status, and transportation resource priorities within the workshop, it achieves globally optimal matching of processing-transportation resource allocation and real-time adaptive balancing of scheduling strategies. Compared to traditional single-agent deep reinforcement learning methods, this invention utilizes a multi-agent cooperative mechanism to decouple high-dimensional complex scheduling tasks into related and parallelizable sub-decision units, overcoming the computational bottleneck and decision lag problems caused by the curse of dimensionality. Simultaneously, combining the truncation optimization characteristics and stable training advantages of the PPO algorithm, it effectively suppresses performance fluctuations caused by excessive policy update amplitudes, ensuring the continuity and robustness of the scheduling scheme under given operating conditions. Ultimately, by reducing equipment idle time, lowering material retention costs, and improving resource utilization, the system significantly enhances decision-making timeliness and scheduling accuracy in complex environments with multiple constraints, thus adapting to the flexible needs of actual production. Attached Figure Description

[0023] Figure 1 The flowchart below shows the flexible job shop scheduling optimization method and system for constrained, multi-type transportation resources according to the present invention. Figure 2 This is the overall flowchart of the MDP of the present invention; Figure 3 This is the training diagram of the centralized training and distributed execution network of the present invention; Figure 4 This is a diagram of the MDP state transition process of the present invention; Figure 5 This is a convergence plot of the network training curve of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] This invention employs a flexible job shop scheduling optimization method and system oriented towards constraints of limited multi-type transportation resources, aiming to address the complexity and strong coupling challenges in integrated transportation-production scheduling scenarios. The system constructs a collaborative decision-making framework composed of three types of functionally differentiated intelligent agents. Each agent undertakes scheduling decision-making tasks in a specific dimension, and task coordination is achieved through cross-agent information interaction and collaborative mechanisms. Its overall architecture is as follows: The core function of the process intelligence agent is to optimize the sequencing and priority selection of unprocessed processes. This involves real-time sensing of the overall system production demand and operational status, combined with key parameters such as delivery date constraints, processing technology requirements, and production resource availability for each process to be processed, to construct a multi-dimensional process priority evaluation system. This system performs a comprehensive quantitative analysis of the priority of all processes. Based on the evaluation results, the process intelligence agent selects the processes that best match the current production capacity and have the highest priority, providing a basis for decision-making in the coordinated scheduling of subsequent processing and transportation.

[0026] The core function of the machine intelligence agent is to allocate the optimal processing equipment to the target process selected by the process intelligence agent. Its decision-making is based on multi-dimensional key indicators, covering the real-time operating status of each processing equipment, processing accuracy, current load rate, process adaptability characteristics, and historical processing efficiency. By comprehensively and quantitatively evaluating the processing adaptability and resource utilization efficiency between the equipment and the target process, the intelligence agent can effectively avoid equipment conflicts and capacity redundancy, ultimately matching the optimal equipment that combines processing quality assurance and production efficiency optimization, ensuring the efficient and coordinated operation of processing and subsequent transportation.

[0027] The core function of the Automated Guided Vehicle (AGV) intelligent agent is to select target processes for the process intelligent agent and complete the optimal transportation scheduling and collaborative connection of processed materials. Its decision-making is based on multiple key indicators, including the AGV's real-time location, load capacity, path congestion status, compatibility with equipment / workstations, and historical transportation efficiency. By comprehensively and quantitatively evaluating the AGV's compatibility with transportation needs and the efficiency of path resource utilization, this intelligent agent can effectively avoid path conflicts and capacity redundancy, matching the optimal AGV that combines transportation timeliness and resource optimization, ensuring efficient and coordinated material flow between processing and subsequent processes / warehousing.

[0028] This technical solution is based on the theoretical framework of DRL and multi-agent cooperative systems. It innovatively adopts the core architecture of centralized training and distributed execution (CTDE). By scientifically decomposing complex and ever-changing scheduling tasks into multiple interrelated sub-problem decision-making units, and relying on the precise local perception capabilities and efficient global coordination mechanisms of each agent, it achieves distributed and efficient solution and dynamic adaptive optimization of sub-problems, and ultimately achieves the global optimal goal of improving scheduling efficiency, maximizing resource utilization, and enhancing the synergy of the entire production process.

[0029] Please see Figures 1-5 A flexible job shop scheduling optimization method and system for constrained limited multi-type transportation resources, specifically including the following steps: S1. Obtain the constraint terms and optimization objective of the flexible job shop scheduling problem with limited multi-type transportation resources, and construct a flexible job shop scheduling model; The Flexible Job Shop Scheduling Problem (FJSP-LTTRs) with finite multi-type transportation resource constraints can be defined as follows: One workpiece Need to Available machines The processing is completed on-site. Due to differences in size, weight, and other dimensions of the workpieces, all AGVs and workpieces are categorized... Before processing, the workpiece needs to be transported to the target machine by an AGV of the matching type. For each process... Only one machine can be selected from its set of available machines for processing, and the processing times of the machines in the set vary. Under the premise of satisfying various constraints such as priority constraints and capacity constraints, the optimization objective of this invention is to minimize the maximum completion time. .

[0030] S1.1 The optimization target is set as the maximum completion time, expressed as: (1); In the formula, The maximum completion time for all workpieces. For process In the machine The processing completion time on the surface For the workpiece index, and ; For machine indexing, and .

[0031] S1.2 Obtain the constraint terms of the flexible job shop scheduling problem under the constraint of finite multi-type transportation resources, including: (1) Uniqueness constraint for workpiece type, ensuring that each job belongs to only one type, as shown in the following expression: (2); In the formula, For a collection of workpieces, A set of workpiece and AGV types. Indexes for workpiece and AGV types, and , Indicates if the workpiece belong Type, then ;otherwise .

[0032] (2) AGV type uniqueness constraint, ensuring that each AGV corresponds to only one type, as shown in the following expression: (3); In the formula, This means if AGV belong Type, then ;otherwise .

[0033] (3) Transportation type matching constraint, to ensure that the execution of transportation tasks meets the consistency requirements between the operation and the AGV type, as shown in the following expression: (4); In the formula, Indicates if process Depend on Type of AGV If transportation is carried out, then ;otherwise ; For AGV index, and .

[0034] (4) Uniqueness constraint for process-AGV allocation, ensuring that any process can only be assigned to a single type of AGV, expressed as: (5); In the formula, For process A collection of compatible AGVs.

[0035] (5) Machine and AGV process continuity constraints ensure that all processes on any machine or AGV assigned a task can form a continuous, acyclic work sequence, expressed as: (6); (7); In the formula, For workpiece The set of processes, , Indicates if process Prior to In the machine Upward processing, ;otherwise ; Indicates if process In the machine Upward processing, ;otherwise ; For process A set of compatible machines; Indicates if process Prior to Depend on Type of AGV If transportation is carried out, then ;otherwise .

[0036] (6) Machine processing sequence consistency constraint, ensuring that the sequence of any two processes on the same machine should be consistent, expressed as: (8); (9); In the formula, It is a very large positive number. Indicate process In the machine The start time of processing.

[0037] (7) AGV transport task sequence constraint, to ensure that the execution order of any two transport tasks undertaken by the same AGV remains consistent, as expressed below: (10); (11); (12).

[0038] (8) Processing time sequence constraints, ensuring that the completion time of any process is not earlier than its start time, as expressed below: (13).

[0039] (9) Processing-transportation start time constraint, ensuring that the processing start time of any process is not earlier than its transportation arrival time, as expressed below: (14).

[0040] In the formula, For process Transported by AGV to the machine Arrival time.

[0041] (10) Machine-AGV path conflict avoidance constraint, ensuring that two processes on the same machine do not have a transportation task, the expression is as follows: (15).

[0042] (11) Transportation-Precedence Completion Time Constraint: Ensures that the transportation arrival time of any process is not earlier than the processing completion time of its preceding process. The expression is as follows: (16); In the formula, For machines With machines The transportation time between them.

[0043] Equation (17) defines the range of values ​​for the binary variables in the model, and its expression is as follows: (17).

[0044] The mathematical parameter variables are defined as shown in Table 1; Table 1: Meaning of Mathematical Indexes, Sets, Constants, and Variables S2. Model the decision-making process of the workshop scheduling model as a multi-agent Markov decision process (MDP), and define three types of agents: process agent, machine agent, and automated guided vehicle (AGV) agent. The process intelligence agent is used for sorting and prioritizing unprocessed processes; The machine intelligence agent is used to allocate the optimal processing equipment to the target process selected by the process intelligence agent; The automated guided vehicle (AGV) intelligent agent is used for the target process selected by the process intelligent agent to complete the optimal transportation scheduling and collaborative connection of processed materials; This invention discloses a multi-agent cooperative scheduling system based on PPO, which models the decision-making process of the flexible job shop scheduling problem with finite multi-type transportation resource constraints as a Markov decision process (MDP). Figure 2 As shown, this MDP mainly consists of five parts: environment, state, action, reward, and state transition; specifically, it includes the following steps: S2.1 Defines the environment for MDP; In the MDP of this invention, the environment is the core carrier for agent decision-making and interaction. It not only needs to define the observable state space of each agent (process agent, machine agent, AGV agent), but also needs to define the observable state space of each agent. and the action space that can be executed in each state It is also necessary to rely on the Markov property and the state transition probability. Complete from the current state To the next state The dynamic transitions, along with the immediate reward output for each agent's "state-action" decision-making, enable real-time rewards. This guides each agent to learn and optimize its strategy, ultimately achieving specific task objectives.

[0045] S2.2. Based on the MDP environment, construct the state space of process intelligent agent, machine intelligent agent and AGV intelligent agent; In MDP, the state space is the core element characterizing the real-time features of the environment and the agent's current situation. It is not only the sole basis for the agent to perceive the environment and choose actions, but it must also satisfy the Markov property: the next state is determined solely by the current state and the actions performed by the agent, independent of historical states. This property significantly simplifies the complexity of decision-making problems. In a complete scheduling process, the system contains several differentiated states, employing... Indicates the current decision-making moment The state in which, This indicates the status characteristics of all current processes. This represents the current state characteristics of all machines. The status characteristics of all AGVs at present are defined as follows: S2.2.1 Define the state characteristics of the process; In this invention It represents the status characteristics of all current processes, specifically including 5 characteristic information items; (1) In Check whether all workpieces have been processed at any given time.

[0046] (2) In The average completion rate of all processes at any given time.

[0047] (3) In The average completion rate of all workpieces at any given time.

[0048] (4) In The standard deviation of the average completion rate of all workpieces at any given time.

[0049] (5) In The average remaining process rate for all workpieces at any given time.

[0050] See Table 1 for details: Table 1: Process Status Characteristics S2.2.2 Define the state characteristics of the machine; In this invention, It represents the current state characteristics of all machines, specifically including three characteristic information items; (1) In The earliest available time for all machines at any given moment.

[0051] (2) In The average utilization rate of all machines at any given time.

[0052] (3) In The standard deviation of the average utilization rate of all machines at any given time.

[0053] See Table 2 for details: Table 2: Machine Status Characteristics S2.2.3 Define the state characteristics of the AGV; In this invention, This indicates the current status characteristics of all AGVs, which specifically includes three feature information items; (1) In The earliest available time for all AGVs at any given moment.

[0054] (2) In The average utilization rate of all AGVs at any given time.

[0055] (3) In The standard deviation of the average utilization rate of all AGVs at any given time.

[0056] See Table 3 for details: S2.3. Based on the MDP environment, construct the motion space of process intelligent agents, machine intelligent agents, and AGV intelligent agents; In the complete scheduling process of this invention, each intelligent agent has an independent action space, specifically including process intelligent agents. Machine intelligence and AGV intelligent body Three categories. Initial time ( In this case, the first process of all workpieces is in a state of waiting to be processed, and the motion space is limited. Depend on It consists of several workpieces; as the scheduling process progresses, the workpiece processes are gradually completed. The range of options will be reduced accordingly. When the action space is empty, it means that all workpieces have been processed and the scheduling process terminates. The action space... Define as an action to be performed Time-compatible machine set, action space Then the corresponding action is executed. A collection of compatible AGVs for different times.

[0057] S2.4. Based on the optimization objective, define three types of agent rewards; In the multi-agent cooperative scheduling system of this invention, due to the adoption of a centralized training and distributed execution CTDE architecture, the common optimization objective of all agents is to minimize the maximum completion time of the workpiece. Therefore, the time interval is defined. The single-step reward function is: .

[0058] Cumulative rewards for the entire round The sum of the single-step rewards at each time step is: .

[0059] In the formula, The start time of scheduling; Therefore, to maximize cumulative rewards, the agent needs to minimize the scheduling termination time. Total completion time This is completely consistent with the core optimization objective of the system of this invention.

[0060] S2.5 Define the state transition process of the MDP; when the three types of agents output a joint action Then, based on the flexible workshop scheduling model, the environment updates the current state of the workshop and calculates and generates the state for the next moment; State transition: State transition represents the transition of the scheduling environment from the current state of the workshop under the joint decision-making of multiple agents. Next state The temporal evolution process. Specifically, at each decision step, when the process, machine, and AGV agent collaboratively establish a joint action including process selection, machine assignment, and transportation scheduling. Subsequently, the environment, acting as the core interaction carrier, executes corresponding processing and transportation tasks according to the flexible workshop scheduling model, and advances the system clock to the next discrete decision moment. During this period, the underlying physical attributes of the system change, including updating the earliest available time windows for each machine and AGV, as well as the real-time processing progress of the workpieces. Then, based on the updated global information, the system recalculates the key features in the state space according to the feature definitions, thereby mapping and generating a new state vector. This closed-loop interaction of "perception-decision-evolution" continues to iterate until all artifacts are completed, marking the end of the scheduling process.

[0061] S3. Based on the proximal strategy optimization of the PPO algorithm and the centralized training and distributed execution CTDE architecture, construct a multi-agent system based on the PPO-CTDE architecture; The multi-agent system includes a policy network of process agents, machine agents, and AGV agents, as well as a centralized value network. The multi-agent scheduling system based on PPO proposed in this invention innovatively adopts a centralized training and distributed execution CTDE architecture. Its complete network training process is as follows: Figure 3 As shown.

[0062] S4. A multi-agent system based on the PPO-CTDE architecture performs network training and optimization in the interaction and update phases to solve the flexible job shop scheduling problem with constraints of limited multi-type transportation resources, and completes the flexible job shop scheduling optimization method and system for constraints of limited multi-type transportation resources. The training process of the multi-agent system of this invention can be clearly divided into two core stages: the interaction stage and the update stage. The interaction stage corresponds to the decentralized execution part in the CTDE architecture where each agent independently perceives the environment and executes actions, while the update stage corresponds to the centralized training part that uses global information to optimize the policy. Specifically, it includes the following steps: S4.1 Interaction Phase: During the interaction between the agent and the environment, the parameters of its policy network remain fixed. The process agent, machine agent, and AGV agent execute distributed decision-making in parallel based on their respective local observations, generating a joint action vector. And it acts on the environment; the environment performs state transitions according to the joint action and the physical constraints of the scheduling model, and feeds back immediate rewards and the next state, generating and storing interaction experience data to the experience storage pool; During the interaction phase, the process intelligence agent, machine intelligence agent, and AGV intelligence agent first rely on... Local state observations are obtained by taking the environmental characteristics at each time point. , and These multidimensional state vectors are then concurrently input into the parameter set. Current policy network In this process, a joint action vector is generated to drive the scheduling process by sampling the probability distribution of the network output. Subsequently, this combined action is directly applied to the environment to execute specific resource allocation and process assignment, causing the system state to dynamically evolve and transition to the next moment's observation. Meanwhile, the environment provides immediate rewards based on preset optimization goals. The system aims to quantitatively assess the effectiveness of the current joint decision-making process. Ultimately, the system will compile this complete Markov decision-making process data. The samples are encapsulated and stored in an experience storage pool to construct a sample set that characterizes the distribution characteristics of the strategy. This provides the necessary data support for subsequent centralized training and network parameter updates based on global value information. This process is repeated until all workpieces are processed.

[0063] like Figure 4 As shown in the figure, this diagram illustrates a 3×3×2 FJSP-LTTRs instance in Time (left image) and The evolution of the scheduling state at time (right figure). In the figure, the colored nodes represent the current... The nodes represent completed processes, while uncolored nodes represent processes awaiting scheduling. Black directed arcs between nodes define process priority constraints, meaning the execution of a subsequent process strictly depends on the completion of its predecessor. Colored directed arcs describe the processing sequence on the same machine, while colored undirected arcs indicate the processing matching relationship between processes and available machines. A colored border for each process indicates that it has been transported by the corresponding type of AGV, while a black border indicates that no AGV has been selected for transport yet. At that moment, the environment executes the joint action (1,2,1), which is the process. Assign to machine And schedule AGVs Transport it to Location; as this process is completed, the system state immediately transitions to... time.

[0064] S4.2 Update Phase: The global state is evaluated based on the experience storage pool. After eliminating the deviation between the evaluation network prediction value and the target state value by calculating and minimizing the mean square error loss function, the advantage function is calculated and the near-end strategy is used to optimize the policy loss objective function of PPO. The policy network parameters of process agent, machine agent and AGV agent are updated synchronously. In the update phase, the fundamental task of this invention is to utilize historical interaction experience data accumulated in the experience storage pool and drive deep optimization of multi-agent collaborative strategies through the guidance of global value information. This process follows the training logic of the Actor-Critic architecture, achieving a closed-loop improvement from environmental perception to decision optimization through the collaborative iteration of the evaluation network (Critic) and the policy network (Actor).

[0065] The expression for calculating the value of the target state is: First, in order to construct an accurate global value evaluation benchmark, the system utilizes an evaluation network to evaluate the joint state. Value fitting is performed. This process aims to eliminate the need to evaluate network predictions. Value of the target state Deviation between; Specifically, this is achieved by minimizing the mean squared error loss function. To achieve this, as shown in formula (18): (18); To ensure the accuracy and stability of the training objectives, the value of the target state... The calculation is based on formula (19). This formula adopts the bootstrapping form of the Bellman equation, which calculates the immediate reward at the current decision moment. Discounted value valuation of the state at the next moment Combined, thus providing parameters The updates provide reliable monitoring signals: (19); After obtaining an effective value assessment, the system further calculates the advantage function. This is used to quantify the superiority or inferiority of the current joint action compared to the average level, thereby guiding the gradient update direction of the policy network. According to formula (20), the advantage function is defined as the actual cumulative reward (including immediate reward and future rewards up to time step). The difference between the discounted return and the current state benchmark value, the advantage function. The expression is as follows: (20); This temporal difference-based computation method effectively reduces the variance of policy gradient estimation, making the training process smoother and more efficient. Ultimately, the policy network parameters... The updates follow the truncation optimization mechanism unique to the PPO algorithm. The system constructs a comprehensive policy loss objective function. As shown in formula (21): (twenty one); The design of this objective function incorporates a dual optimization logic: on the one hand, the truncation term... Forced probability ratio of new and old strategies Restricted to a safe area Internally, this avoids strategy collapse caused by excessively large single update steps; on the other hand, the entropy regularization term introduced in the formula... By leveraging the uncertainty of reward action distribution, agents are continuously encouraged to explore diverse scheduling actions in the early stages of training, thereby effectively preventing the system from converging to a local optimum prematurely and ensuring the robustness and global optimization capability of the multi-agent collaborative strategy.

[0066] The termination condition is: the selectable action space of the process agent is empty, indicating that all workpieces have been processed. Figure 5 The training curve is shown on a 20×6×4 instance. The pseudocode for the specific training steps is shown in Algorithm 1: Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A flexible job shop scheduling optimization method for constraints of limited multi-type transportation resources, characterized in that, Includes the following steps: S1. Obtain the constraint terms and optimization objective of the flexible job shop scheduling problem with limited multi-type transportation resources, and construct a flexible job shop scheduling model; S2. Model the decision-making process of the workshop scheduling model as a multi-agent Markov decision process (MDP), and define three types of agents: process agent, machine agent, and automated guided vehicle (AGV) agent. The process intelligence agent is used for sorting and prioritizing unprocessed processes; The machine intelligence agent is used to allocate the optimal processing equipment to the target process selected by the process intelligence agent; The automated guided vehicle (AGV) intelligent agent is used for the target process selected by the process intelligent agent to complete the optimal transportation scheduling and collaborative connection of processed materials; S3. Based on the proximal strategy optimization of the PPO algorithm and the centralized training and distributed execution CTDE architecture, construct a multi-agent system based on the PPO-CTDE architecture; S4. A multi-agent system based on the PPO-CTDE architecture performs network training and optimization in the interaction and update phases to solve the flexible job shop scheduling problem with constraints of limited multi-type transportation resources, and completes a flexible job shop scheduling optimization method and system for constraints of limited multi-type transportation resources.

2. The flexible job shop scheduling optimization method for constraints of limited multi-type transportation resources as described in claim 1, characterized in that, S1 specifically includes the following steps: S1.1 The optimization target is set as the maximum completion time, expressed as: (1)； In the formula, The maximum completion time for all workpieces. For process In the machine The processing completion time on the surface For the workpiece index, and ; For machine indexing, and ; S1.2 Obtain the constraint terms of the flexible job shop scheduling problem under the constraint of finite multi-type transportation resources, including: (1) Uniqueness constraint of workpiece type, ensuring that each workpiece belongs to only one type; (2) AGV type uniqueness constraint to ensure that each AGV corresponds to only one type; (3) Transportation type matching constraints to ensure that the execution of transportation tasks meets the consistency requirements between the workpiece and the AGV type; (4) Uniqueness constraint for process-AGV allocation, ensuring that any process can only be assigned to a single type of AGV; (5) Machine and AGV process continuity constraints ensure that all processes on any machine or AGV assigned a task can form a continuous, acyclic work sequence. (6) Consistency constraint of machine processing sequence, to ensure that the sequence of any two processes on the same machine should be consistent; (7) AGV transportation task sequence constraints to ensure that any two transportation tasks undertaken by the same AGV should be consistent; (8) Processing time sequence constraints to ensure that the processing completion time of any process is not earlier than its processing start time; (9) Processing-transportation start time constraints to ensure that the processing start time of any process is not earlier than its transportation arrival time; (10) Machine-AGV path conflict avoidance constraint to ensure that two processes processed on the same machine will not have a transportation task; (11) Transportation - Precedence completion time constraint, ensuring that the transportation arrival time of any process shall not be earlier than the processing completion time of its predecessor process.

3. The flexible job shop scheduling optimization method for constraints of limited multi-type transportation resources as described in claim 1, characterized in that, S2 specifically includes the following steps: S2.1 Defines the environment for MDP; S2.

2. Based on the MDP environment, construct the state space of process intelligent agent, machine intelligent agent and AGV intelligent agent; S2.

3. Based on the MDP environment, construct the motion space of process intelligent agents, machine intelligent agents, and AGV intelligent agents; S2.

4. Based on the optimization objective, define three types of agent rewards; Define time The single-step reward function is: Cumulative rewards for the entire round The sum of the single-step rewards at each time step is: In the formula, The start time of scheduling Indicates the current decision-making moment state, The total number of decision steps. The maximum completion time for all workpieces. This is the time when the scheduling ends.

4. The flexible job shop scheduling optimization method for constraints of limited multi-type transportation resources according to claim 3, characterized in that, S2.2 specifically includes the following steps: S2.2.1 Define the state characteristics of the process, specifically including 5 characteristic information items: (1) Whether all workpieces have been processed at the current decision-making moment; (2) The average completion rate of all processes at the current decision-making moment; (3) The average completion rate of all workpieces at the current decision-making moment; (4) The standard deviation of the average completion rate of all workpieces at the current decision-making time; (5) The average remaining process rate of all workpieces at the current decision time; S2.2.2 Define the machine's state characteristics, specifically including three characteristic information items: (1) The earliest available time of all machines at the current decision moment; (2) The average utilization rate of all machines at the current decision-making moment; (3) The standard deviation of the average utilization rate of all machines at the current decision-making moment; S2.2.3 Define the state characteristics of the AGV, specifically including three characteristic information items: (1) The earliest available time of all AGVs at the current decision-making moment; (2) The average utilization rate of all AGVs at the current decision-making moment; (3) The standard deviation of the average utilization rate of all AGVs at the current decision time.

5. The flexible job shop scheduling optimization method for constraints of limited multi-type transportation resources according to claim 2, characterized in that, S4 specifically includes the following steps: S4.1 Interaction Phase: During the interaction between the agent and the environment, the parameters of its policy network remain fixed. The process agent, machine agent, and AGV agent execute distributed decision-making in parallel based on their respective local observations, generating a joint action vector. And it acts on the environment; the environment performs state transitions according to the joint action and the physical constraints of the scheduling model, and feeds back immediate rewards and the next state, generating and storing interaction experience data to the experience storage pool; S4.2 Update Phase: The global state is evaluated based on the experience storage pool. After eliminating the deviation between the evaluation network prediction value and the target state value by calculating and minimizing the mean square error loss function, the advantage function is calculated and the near-end strategy is used to optimize the policy loss objective function of PPO. The policy network parameters of the process agent, machine agent and AGV agent are updated simultaneously.

6. The flexible job shop scheduling optimization method for constraints of limited multi-type transportation resources according to claim 5, characterized in that, In S4.2, the expression for minimizing the mean squared error loss function is: In the formula, To evaluate the network predictions, For the target state value, For expectation operators; The expression for the policy loss objective function is: In the formula, For truncation terms, The probability ratio between the old and new strategies. For the safe zone, For entropy regularization, This is the dominant function.

7. The flexible job shop scheduling optimization method for constraints of limited multi-type transportation resources according to claim 1, characterized in that, In step S4, the termination condition for network training and optimization is: the selectable action space of the process agent is empty, indicating that all workpieces have been processed.

8. A flexible job shop scheduling optimization system for constraints of limited multi-type transportation resources, characterized in that, This method is used to run a flexible job shop scheduling optimization method for constrained, multi-type transportation resources as described in any one of claims 1-7.

Images