Fms scheduling and resource configuration joint optimization method, system, device and medium

Abstract

The application discloses a FMS scheduling and resource configuration joint optimization method, system, device and medium, belongs to the field of manufacturing system optimization, adopts S 3 The PR network constructs a Petri network model of the FMS, and characterizes workpieces and processes in the FMS; a scheduling decision module executes an improved Dyna-Q algorithm under a machine time change environment based on resource configuration, obtains a maximum completion time set of multiple groups of time samples; a resource configuration decision module analyzes the set, calculates a reward function value, explores machine number configuration under a cost constraint, and generates a new scheme to feed back to the scheduling decision module; the two modules are iteratively optimized in interaction to meet resource and cost constraints, minimize the maximum completion time of the system, and output resource configuration and scheduling results to generate an executable scheme according to workpiece process start time sorting, thereby improving the quality and efficiency of scheduling and resource configuration.

Description

Technical Field

[0001] This invention belongs to the field of manufacturing system optimization, specifically involving a method, system, equipment, and medium for joint optimization of FMS scheduling and resource allocation. Background Technology

[0002] Flexible Manufacturing Systems (FMS) are a key means of addressing market diversification and order uncertainty. FMS involves multiple processes, multiple workpieces, and multiple types of equipment. Resources are limited, and concurrency and synchronization relationships exist; therefore, scheduling and resource allocation are core issues affecting system performance. Petri nets, as a modeling tool for discrete event systems, are widely used to describe and solve scheduling problems in manufacturing systems because they can explicitly express concurrency, resource contention, and synchronization constraints. Traditional optimization methods (heuristics, genetic algorithms, ant colony optimization, etc.) can achieve good solutions for small- to medium-scale problems, but they often get trapped in local optima or struggle to maintain robustness when state space explodes or time / process parameters change dynamically.

[0003] In recent years, reinforcement learning has been introduced into manufacturing scheduling research. Q-learning (action value function learning) methods gradually improve policies through interaction with the environment, offering advantages such as not requiring strong prior models and possessing online adaptability. However, in large-scale manufacturing scheduling systems (FMS), reinforcement learning faces several challenges: First, in large-scale manufacturing systems, numerous deadlock states can affect the agent's exploration of the optimal scheduling path, thus reducing the quality of the scheduling solution. Second, in practical manufacturing scheduling problems, resource allocation also significantly impacts the scheduling results, and evaluating a resource allocation requires consuming complete scheduling solution resources, severely affecting overall solution efficiency. Third, machine processing time in practical manufacturing systems is not constant but often fluctuates within a certain range (randomness or influenced by workpieces / tools / environment), causing policies trained using deterministic models to perform poorly in real-world scenarios. Traditional reinforcement learning and planning based on a single deterministic time model cannot effectively address this time uncertainty.

[0004] Therefore, a robust training and evaluation mechanism accommodating uncertainty needs to be introduced into the methodology. This invention proposes describing machine processing time using time intervals and explicitly incorporating uncertainty into the training and evaluation process through scenario sampling: At the scheduling end, the internal model of Dyna-Q (Dynamic Q-learning) is changed from a single-value mapping to a sample pool model. During the priority replay and planning phases, non-deadlock samples from different sampling scenarios are replayed to estimate the expected reward of the action corresponding to each state. At the resource allocation end, each candidate configuration is sampled and evaluated multiple times, and statistics are cached to reduce costly repeated scheduling calls and obtain robust configuration comparison metrics. Furthermore, the two-layer loop of outer scenario sampling and inner training can cover more time scenarios and improve the policy's generalization ability to time fluctuations.

[0005] The aforementioned improvements aim to address three shortcomings: first, to break the separation of scheduling and resource allocation in optimization, and to construct a joint training framework for scheduling and resource allocation in flexible manufacturing systems that considers the uncertainty of processing or transportation time during scheduling; second, to improve the quality of reinforcement learning in solving scheduling problems in large-scale flexible manufacturing systems with numerous deadlocks; and third, to significantly reduce the computational resources required for resource allocation and to provide resource allocation schemes that consider both the maximum expected completion time and risk assessment under budget constraints. Therefore, in real-world flexible manufacturing scenarios with time fluctuations, there is an urgent need for a reinforcement learning joint optimization method that combines Petri net modeling, sample pool planning, outer scenario sampling, and multi-sample configuration evaluation to improve the overall system performance and decision reliability. Summary of the Invention

[0006] The purpose of this invention is to overcome the complex scheduling problem of large-scale flexible manufacturing systems, and to propose a joint optimization method, system, equipment and medium for FMS scheduling and resource allocation.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a joint optimization method for FMS scheduling and resource allocation, comprising the following steps:

[0009] Based on the actual production relationship of the Flexible Manufacturing System (FMS), an S-type system with warehouse delay is adopted. 3 PR network constructs Petri net model of FMS, obtains S with place delay 3 PR network model, S with library delay 3 The PR net model represents the set of workpieces, process sequence, resource set, and time characteristics of each process in FMS. The resource set includes processing machines and transport robots.

[0010] The scheduling decision module is activated, taking into account the S-type delay of the warehouse. 3 The PR network model, based on the initial or current resource allocation scheme, executes an improved Dyna-Q algorithm under varying machine time conditions to obtain the maximum completion time set under multiple time samples. The scheduling decision module includes scheduling strategy learning and scheduling strategy application. The scheduling strategy learning includes an outer loop and an inner loop. The outer loop randomly samples the processing time interval and transportation time interval of each process to obtain time samples. The inner loop, based on the time samples, completes the scheduling strategy learning through the improved Dyna-Q algorithm, and then sends the scheduling decision results corresponding to each set of time samples to the resource allocation decision module through scheduling strategy application.

[0011] The resource allocation decision module is activated to analyze and evaluate the scheduling decision results set under multiple time samples, calculate the reward function value, explore the number of machines under cost constraints, generate a new resource allocation scheme and feed it back to the scheduling decision module.

[0012] Based on the new resource allocation scheme, the scheduling strategy learning and application of the scheduling decision module are repeated to obtain a new set of scheduling decision results under multiple time samples. Based on the new set of scheduling decision results under multiple time samples, the resource allocation decision module is repeatedly started, and joint optimization iteration is performed until the preset termination condition is met. The goal of the joint optimization iteration is to minimize the maximum system completion time under the premise of satisfying resource and cost constraints.

[0013] After the joint optimization iteration is completed, the final resource allocation and scheduling results are output. The time axis is sorted according to the start time of each workpiece and each process to generate an executable production scheduling plan.

[0014] Furthermore, the specific process of learning the scheduling strategy in the scheduling decision module is as follows:

[0015] Construct the nominal system state and the sampled system state; both the nominal system state and the sampled system state contain Petri net state identifiers and corresponding remaining processing time. The nominal system state corresponds to the nominal remaining time, and the sampled system state corresponds to the actual remaining processing time after sampling the current machine processing time and after each action selection; at the same time, set the initial state and target state of the system.

[0016] Obtain the initial or current resource configuration scheme output by the resource configuration decision module, and perform outer loop training and inner loop training of the scheduling decision module. The outer loop training includes sampling the processing time interval of each machine, setting the total number of outer loops, using a set of processing time sampling samples for each outer loop, and inputting the processing time sampling samples into the inner loop for sample training.

[0017] The inner loop training includes sampling samples and system information based on the current processing time, learning and applying the scheduling strategy of the samples through the improved Dyna-Q algorithm, obtaining the scheduling strategy and results of the scheduling decision module, and retaining the currently trained scheduling strategy and results of the scheduling decision module after the inner loop ends and proceeding to the next outer loop training.

[0018] The specific process of applying the scheduling strategy in the scheduling decision module is as follows:

[0019] After the outer loop training and inner loop training of the scheduling decision module are completed, the Q_s table of the trained scheduling decision module is sampled and deployed multiple times, and the results obtained from the multiple sampling deployments are recorded. After the sampling deployment is completed, the final Q_s table and the set of maximum completion times for each time sample are obtained.

[0020] Furthermore, the improved Dyna-Q algorithm includes the following steps:

[0021] Step 1: Initialize scheduling parameters, scheduling policy table, scheduling replay list, scheduling transfer list, and scheduling predecessor list;

[0022] Step 2: Initialize the identifier to obtain the current identifier, and initialize the nominal remaining time and the actual remaining processing time after the action selection;

[0023] Step 3: Based on the current identifier obtained in Step 2, determine the set of enabling transitions under the Petri net, and adopt an improved global exploration rate. The greedy strategy selects the transition enable as the action in the current state, obtains the new identifier and the corresponding nominal state, sampled state and new state. The new state is the set of the nominal state and the sampled state, and generates state transition samples.

[0024] Step 4: Determine the state type of the new identifier obtained in Step 3, and calculate the immediate reward based on the state type; the state types include deadlock state, target identifier state and general state, and different state types correspond to different immediate reward calculation methods;

[0025] Step 5: Record the state transition sample generated in Step 3 into the scheduling transition list, and at the same time record the current state as the predecessor state of the new state obtained in Step 3, and store it in the scheduling predecessor list.

[0026] Step 6: Reuse the immediate reward from Step 4 and the new state from Step 3 to calculate the timing difference error. If the new state is not a deadlock state and the timing difference error meets the preset conditions, store the corresponding experience in the scheduling replay list, and then use the improved adaptive decay learning rate to update the scheduling strategy table.

[0027] Step 7: Perform virtual replay a preset number of times. Each time, extract the highest priority experience from the scheduling replay list. Based on the scheduling transfer list constructed in Step 5, determine the new state corresponding to the highest priority experience. Update the policy value corresponding to the highest priority experience in conjunction with the scheduling policy table. At the same time, process the timing difference error of the predecessor state based on the scheduling predecessor list in Step 5, and store the experience that meets the conditions into the scheduling replay list.

[0028] Step 8: After updating the nominal state and sampled state obtained in Step 3, return to Step 3 and continue the loop until the new state is the target state or deadlock state to end this round of optimization; repeat the above optimization process until the preset total number of optimization rounds is reached to complete the outer loop optimization;

[0029] Step 9: After completing the training of all outer loops in Step 8, the trained scheduling strategy table is sampled and deployed a preset number of times; starting from the initial state, the action with the optimal strategy value is selected for execution, the reward of each action is recorded and accumulated until the target state is reached, the scheduling strategy table is kept frozen during the sampling and deployment process, and finally the sampling and deployment result set and the scheduling strategy table are output.

[0030] Furthermore, the improved global exploration rate Greedy strategy, specifically:

[0031] Set global exploration rate initial value Global exploration rate final value ;

[0032] During the outer training loop of each scenario sampling, the internal global exploration rate... Employing an exponential decay strategy, the inner layer... The exploration rate of each round , It is the attenuation factor;

[0033] After completing the inner loop, a temperature recovery process is performed when entering the next outer loop scenario sampling. This process is repeated whenever a new scenario is entered. The cycle needs a brief rebound. Exploring new scenarios ,in This is the temperature difference. This is the exploration rate at the end of the previous inner loop;

[0034] The adaptive decay learning rate is shown in the following formula:

[0035]

[0036] in, This represents the adaptive decay learning rate. r represents the number of iterations in the outer loop. Represents the number of inner loop iterations. This represents the initial learning rate or the learning rate at the end of the previous inner / outer loop. and This represents the decay coefficient of the learning rate in the outer and inner loops.

[0037] The priority of the playback list in the improved Dyna-Q algorithm takes into account timing difference errors. Simultaneously, a decay weight is added, and the priority of each experience entry is calculated using the decay weight. The experience entry with the highest priority is placed at the top of the min-heap replay list. The decay weight is shown in the following formula:

[0038]

[0039] in, For decay weights, This is the forgetting rate coefficient. and This refers to the current outer layer number and the outer layer number corresponding to the experience entry.

[0040] Furthermore, the resource allocation learning process of the resource allocation decision module includes the following steps:

[0041] Step 01: Initialize the resource configuration strategy table, configuration replay list, configuration state transition list, and configuration predecessor list, and set the training parameters to obtain the initial resource configuration;

[0042] Step 02: After initialization is complete, enter the loop iteration, taking the initial resource configuration as the initial state. The initial state represents the quantity configuration of various processing equipment and transportation equipment.

[0043] Step 03: Based on the resource configuration corresponding to the current initial state, solve the scheduling decision module to obtain the set of scheduling results under the current resource configuration;

[0044] Step 04: After obtaining the scheduling result set, determine the set of legal actions for the current state. The set of legal actions is subject to cost budget constraints. Select and execute actions based on a greedy strategy to obtain a new state.

[0045] Step 05: After obtaining the new state, calculate the evaluation index and reward corresponding to the new state; if the resource allocation corresponding to the new state exceeds the cost budget, apply a penalty and revert the state to the original state, then return to step 4 to select an action again;

[0046] Step 06: After completing the reward calculation and state rollback processing, record the state transition relationships by configuring the state transition list and the predecessor list;

[0047] Step 07: After recording the state transition relationship, update the Q value; if the timing difference error exceeds the preset threshold, push the state action information into the configuration playback list;

[0048] Step 08: After completing the Q-value update and pushing the configuration replay list entries, perform a preset number of loop replays. Each time, the highest priority entry is popped from the configuration replay list. The subsequent state is determined based on the configuration state transition list. The Q-value corresponding to the optimal action in the subsequent state is found by combining the configuration strategy table, and the Q-value of the current entry is updated. At the same time, the set of predecessor states of the current state is determined based on the configuration predecessor list. The timing difference error of each predecessor state is calculated. The predecessor state information that meets the threshold condition is pushed into the configuration replay list until the preset number of loop replays is completed.

[0049] Step 09: After completing the preset number of loop replays, update the current state to the new state, accumulate the training rounds, and return to Step 3 to continue iterating until the preset total number of optimization rounds is reached, and the optimization ends;

[0050] Step 010: After optimization, the deployment phase begins. Based on the optimized strategy, the final resource configuration is obtained from the initial configuration using a greedy strategy. The scheduling decision module is then executed again based on the final resource configuration to obtain the final scheduling result.

[0051] Furthermore, the calculation process of the evaluation indicators in the resource allocation decision-making module includes:

[0052] The sample mean and sample standard deviation are calculated based on the scheduling result set. Based on the sample mean and sample standard deviation, a comprehensive evaluation index is constructed using the comprehensive evaluation index formula.

[0053] The sample mean and sample standard deviation are calculated based on the scheduling result set as shown in the following formula:

[0054]

[0055]

[0056] in, The sample mean. The standard deviation of the sample is 1. For the number of sampling deployments, For current resource allocation The next The scheduling results for each sample;

[0057] The formula for the comprehensive evaluation index is as follows:

[0058]

[0059] in, As a comprehensive evaluation indicator, As an uncertainty penalty factor, ;

[0060] When outputting the final resource configuration and scheduling results, select a set of candidate configurations with the top Q values ​​after training, re-solve the scheduling for each configuration, and select the configuration that meets the cost budget and has the shortest maximum completion time as the final resource configuration.

[0061] Secondly, the present invention provides an FMS scheduling and resource allocation joint optimization system, characterized in that it includes:

[0062] The model building module is used for the actual production relationships based on the Flexible Manufacturing System (FMS), employing S with warehouse delay. 3 PR network constructs Petri net model of FMS, obtains S with place delay 3 PR network model, S with library delay 3 The PR net model represents the set of workpieces, process sequence, resource set, and time characteristics of each process in FMS. The resource set includes processing machines and transport robots.

[0063] The scheduling decision module is used to combine the S with the warehouse delay. 3 The PR network model, based on the initial or current resource allocation scheme, executes an improved Dyna-Q algorithm under varying machine time conditions to obtain the maximum completion time set under multiple time samples. The scheduling decision module includes scheduling strategy learning and scheduling strategy application. The scheduling strategy learning includes an outer loop and an inner loop. The outer loop randomly samples the processing time interval and transportation time interval of each process to obtain time samples. The inner loop, based on the time samples, completes the scheduling strategy learning through the improved Dyna-Q algorithm, and then sends the scheduling decision results corresponding to each set of time samples to the resource allocation decision module through scheduling strategy application.

[0064] The resource allocation decision module is used to analyze and evaluate the set of scheduling decision results under multiple time samples, calculate the reward function value, explore the number of machines under cost constraints, generate new resource allocation schemes and feed them back to the scheduling decision module.

[0065] The scheduling and resource allocation joint optimization module is used to repeatedly learn and apply the scheduling strategy of the scheduling decision module based on the new resource allocation scheme, and obtain a new set of scheduling decision results under multiple time samples; based on the new set of scheduling decision results under multiple time samples, the resource allocation decision module is repeatedly started, and the joint optimization iteration is carried out until the preset termination condition is met; the goal of the joint optimization iteration is to minimize the maximum system completion time under the premise of satisfying resource and cost constraints.

[0066] The production scheduling scheme generation module is used to output the final resource configuration and scheduling results after the joint optimization iteration is completed. It sorts the time axis according to the start time of each workpiece and each process, and generates an executable production scheduling scheme.

[0067] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a joint optimization method for FMS scheduling and resource allocation.

[0068] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a joint optimization method for FMS scheduling and resource allocation.

[0069] Compared with the prior art, the present invention has the following beneficial technical effects:

[0070] This invention proposes a joint optimization method for FMS scheduling and resource allocation. To further improve scheduling efficiency and result quality, and to address the issue of requiring a costly entire scheduling decision module for evaluating resource allocation, this invention introduces an improved Dyna-Q algorithm for both scheduling and resource allocation. Dyna-Q is a classic reinforcement learning framework. Compared to traditional algorithms such as metaheuristics and common reinforcement learning algorithms like Q-learning, it updates the value function using real samples generated by the environment and stores these samples in the internal model, "virtually replaying" them several times for further updates. It is better suited to solving problems with unknown environmental parameters (such as varying machine processing time or unknown machine numbers), and can significantly improve training efficiency and optimization strategies. It can exchange a small number of expensive samples for a large number of virtual updates, thus significantly improving learning efficiency and robustness in real-world problems where samples are scarce or simulation is expensive. This is combined with the S-curve of the Petri net with the time delay of the library. 3 The PR (Systems of Simple Sequential Processes with Resources) network modeling approach, combined with the introduction of an improved Dyna-Q reinforcement learning algorithm, effectively solves the large-scale FMS scheduling problem under varying machine processing times. It also significantly improves the quality of scheduling solutions, optimizes resource allocation schemes, and enhances configuration training efficiency. This invention rationally and efficiently addresses the joint optimization problem of scheduling and resource allocation in complex environments, reducing the maximum completion time of system machine processing and improving resource utilization and scheduling efficiency under cost constraints. It is particularly suitable for optimization scenarios in multi-task complex flexible manufacturing systems. Attached Figure Description

[0071] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of the invention in any way. Furthermore, the shapes and proportions of the components in the drawings are merely illustrative to aid in understanding the invention and do not specifically limit the shapes and proportions of the components.

[0072] In the attached diagram:

[0073] Figure 1 This is a flowchart of the FMS scheduling and resource allocation joint optimization method of the present invention.

[0074] Figure 2 This is a simplified structural diagram of the FMS scheduling and resource allocation joint optimization system of the present invention.

[0075] Figure 3 This is a schematic diagram of an electronic device for the FMS scheduling and resource allocation joint optimization method of the present invention.

[0076] Figure 4 This is a flowchart of the joint optimization method for FMS scheduling and resource configuration in an embodiment of the present invention.

[0077] Figure 5 This is a simplified structural diagram of the FMS scheduling and resource allocation joint optimization system in an embodiment of the present invention.

[0078] Figure 6 This is a flowchart of the joint optimization method for FMS scheduling and resource configuration in an embodiment of the present invention.

[0079] Figure 7 The production requirements of the flexible manufacturing system in this embodiment of the invention are as follows.

[0080] Figure 8 This is a schematic diagram of the processing of the flexible manufacturing system in an embodiment of the present invention.

[0081] Figure 9 This is a Petri net model diagram of the workshop environment of the flexible manufacturing system in an embodiment of the present invention. .

[0082] Figure 10 This is a flowchart of a multi-agent reinforcement learning process based on Petri nets with place delay in an embodiment of the present invention.

[0083] Figure 11 This is a Gantt chart showing the scheduling results of the flexible manufacturing system in this embodiment of the invention. Detailed Implementation

[0084] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0085] Example 1

[0086] See Figure 1 The joint optimization method for FMS scheduling and resource allocation includes the following steps:

[0087] Based on the actual production relationship of the Flexible Manufacturing System (FMS), an S-type system with warehouse delay is adopted. 3 PR network constructs Petri net model of FMS, obtains S with place delay 3 PR network model, S with library delay 3 The PR net model represents the set of workpieces, process sequence, resource set, and time characteristics of each process in FMS. The resource set includes processing machines and transport robots.

[0088] The scheduling decision module is activated, taking into account the S-type delay of the warehouse. 3 The PR network model, based on the initial or current resource allocation scheme, executes an improved Dyna-Q algorithm under varying machine time conditions to obtain the maximum completion time set under multiple time samples. The scheduling decision module includes scheduling strategy learning and scheduling strategy application. The scheduling strategy learning includes an outer loop and an inner loop. The outer loop randomly samples the processing time interval and transportation time interval of each process to obtain time samples. The inner loop, based on the time samples, completes the scheduling strategy learning through the improved Dyna-Q algorithm, and then sends the scheduling decision results corresponding to each set of time samples to the resource allocation decision module through scheduling strategy application.

[0089] The resource allocation decision module is activated to analyze and evaluate the scheduling decision results set under multiple time samples, calculate the reward function value, explore the number of machines under cost constraints, generate a new resource allocation scheme and feed it back to the scheduling decision module.

[0090] Based on the new resource allocation scheme, the scheduling strategy learning and application of the scheduling decision module are repeated to obtain a new set of scheduling decision results under multiple time samples. Based on the new set of scheduling decision results under multiple time samples, the resource allocation decision module is repeatedly started, and joint optimization iteration is performed until the preset termination condition is met. The goal of the joint optimization iteration is to minimize the maximum system completion time under the premise of satisfying resource and cost constraints.

[0091] After the joint optimization iteration is completed, the final resource allocation and scheduling results are output. The time axis is sorted according to the start time of each workpiece and each process to generate an executable production scheduling plan.

[0092] This embodiment demonstrates outstanding modeling accuracy, including S with library delay. 3 The PR network comprehensively represents the characteristics of workpieces, processes, processing / transportation resources, and time, closely aligning with the actual production relationships of a Flexible Manufacturing System (FMS) and providing reliable model support for optimization. It exhibits strong dynamic adaptability; the improved Dyna-Q algorithm is trained under varying machine time environments, covering multiple time samples to ensure the robustness of the scheduling scheme in the face of production fluctuations. Collaborative optimization is highly efficient, with dual-process interactive iteration. Resource allocation explores the number of machines under cost constraints, while scheduling simultaneously optimizes process arrangements, achieving deep collaboration between resource allocation and scheduling. This satisfies constraints while minimizing the system's maximum completion time. The output time-axis sorted production scheduling scheme is directly executable, highly implementable, and significantly improves FMS production efficiency, reduces resource waste and production costs, providing strong support for the efficient operation of flexible manufacturing systems.

[0093] Example 2

[0094] See Figure 2 The FMS scheduling and resource allocation joint optimization system includes:

[0095] The model building module is used for the actual production relationships based on the Flexible Manufacturing System (FMS), employing S with warehouse delay. 3 PR network constructs Petri net model of FMS, obtains S with place delay 3 PR network model, S with library delay 3 The PR net model represents the set of workpieces, process sequence, resource set, and time characteristics of each process in FMS. The resource set includes processing machines and transport robots.

[0096] The scheduling decision module is used to combine the S with the warehouse delay. 3 The PR network model, based on the initial or current resource allocation scheme, executes an improved Dyna-Q algorithm under varying machine time conditions to obtain the maximum completion time set under multiple time samples. The scheduling decision module includes scheduling strategy learning and scheduling strategy application. The scheduling strategy learning includes an outer loop and an inner loop. The outer loop randomly samples the processing time interval and transportation time interval of each process to obtain time samples. The inner loop, based on the time samples, completes the scheduling strategy learning through the improved Dyna-Q algorithm, and then sends the scheduling decision results corresponding to each set of time samples to the resource allocation decision module through scheduling strategy application.

[0097] The resource allocation decision module is used to analyze and evaluate the set of scheduling decision results under multiple time samples, calculate the reward function value, explore the number of machines under cost constraints, generate new resource allocation schemes and feed them back to the scheduling decision module.

[0098] The scheduling and resource allocation joint optimization module is used to repeatedly learn and apply the scheduling strategy of the scheduling decision module based on the new resource allocation scheme, and obtain a new set of scheduling decision results under multiple time samples; based on the new set of scheduling decision results under multiple time samples, the resource allocation decision module is repeatedly started, and the joint optimization iteration is carried out until the preset termination condition is met; the goal of the joint optimization iteration is to minimize the maximum system completion time under the premise of satisfying resource and cost constraints.

[0099] The production scheduling scheme generation module is used to output the final resource configuration and scheduling results after the joint optimization iteration is completed. It sorts the time axis according to the start time of each workpiece and each process, and generates an executable production scheduling scheme.

[0100] Example 3

[0101] See Figure 3 An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a joint optimization method for FMS scheduling and resource allocation when executing the computer program.

[0102] Example 4

[0103] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a joint optimization method for FMS scheduling and resource allocation.

[0104] Example 5

[0105] See Figure 4 The joint optimization method for FMS scheduling and resource allocation includes the following steps:

[0106] Step 1: Based on the production relations of the actual flexible manufacturing system, model the flexible manufacturing system using a Petri net with warehouse-place delay. Establish the S-type model with warehouse-place delay under the Petri net of the flexible manufacturing system according to actual production requirements. 3 PR network model; a practical flexible manufacturing system mainly includes various types of workpieces to be processed, multiple processing steps, and several parallel processing devices and shared resources. Based on actual production requirements, the workpieces to be processed are grouped together... To represent, where The first part represents the flexible manufacturing system. There are a total of [number] types of production workpieces. Type of production workpiece.

[0107] Each type of workpiece Consists of a set of ordered processes composition:

[0108]

[0109] in, Indicates the workpiece being processed The One process, This represents the number of operations required to process the workpiece. The operations include machining by the processing machine and transportation by the transport robot.

[0110] Resource distribution in flexible manufacturing systems mainly includes two types: processing machines. and transport vehicle robots Its processing machines are used for It is said that the transport robot is used Indicates. Processing machinery. The transport robot is responsible for completing each processing step of each type of workpiece. Responsible for transferring the workpiece to the next processing machine. The number of each type of processing machine is... The number of each type of transport robot is The resource set of the entire flexible manufacturing system is represented as follows: .

[0111] Processing machines and transport robots both require a certain amount of processing and transport time during the processing and transportation processes. Each process... Available in candidate resource set Execute on any machine in the network.

[0112] Its processing machinery The processing time is a range of values:

[0113]

[0114] Meanwhile, the processing steps require the use of transport robots before and after execution. The corresponding transportation time after loading or transshipment is completed is as follows:

[0115]

[0116] Each processed workpiece The procedures must be completed in the given order:

[0117]

[0118] Process The execution must meet the following conditions:

[0119] 1. Previous process Completed;

[0120] 2. Required processing machines Available;

[0121] 3. The assigned transport robot idle.

[0122] The completion time for each machined workpiece is The maximum completion time (Makespan) of the system is:

[0123]

[0124] The total resource cost of the system is subject to the budget. constraint:

[0125]

[0126] The ultimate optimization goal is to minimize the maximum completion time while satisfying resource and cost constraints.

[0127]

[0128] To better describe the concurrent processing, resource sharing, and mutual exclusion relationships in flexible manufacturing systems, this invention employs a Cubic-Position Delay Petri Net (SPON). 3 The PR model models this process. Through the formal definitions of places, transitions, and identifiers, this model can accurately depict the workpiece and resource allocation process in a flexible manufacturing system.

[0129] S 3 PR model is defined as Among them, the collection of warehouses , For the system to correspond to S 3 The number of places in the PR model represents the system state or resources, including active places, resource places, starting places, and ending places:

[0130]

[0131] Among them, activity warehouse This indicates the intermediate state of a workpiece during a specific processing stage; the resource library contains... The token number in the table represents the available number of processing machines or transport robots, starting from the warehouse. This indicates that the current workpiece has not yet been put into processing, and the inventory period ends. It represents the state of the workpiece after processing.

[0132] Set of Changes , For the system to correspond to S 3 The number of transitions in a PR model represents the occurrence of system events or actions.

[0133] For flow relationship, The weight functions correspond to the order of each process in the system and the number of resources required, respectively.

[0134] Processing steps and the release and use of resources are characterized by the enabling emission of transitions, and the state of the system is identified by flags. It indicates that when changes occur Enabled (i.e., when all input libraries of the transition contain at least one token):

[0135]

[0136] This allows for transition-enabled transmission and updates to the system state:

[0137]

[0138] in Representing changes in Petri nets The preceding correlation matrix, Representing changes in Petri nets The post-transition correlation matrix. When a transition in the system is enabled, the weight values ​​on the corresponding directed arcs need to be consumed from all its preceding places, and the weight values ​​on the corresponding directed arcs need to be added to its subsequent places. This is used to characterize the completion and commencement of processing of the machine or transport robot in the actual flexible manufacturing system.

[0139] For each workpiece Each process sequence subnet needs to be constructed based on its processing sequence. After construction, a resource subnet needs to be constructed based on resource sharing within the processing sequence. Finally, these subnets are merged to complete the modeling. For the workpiece... Its actual processing sequence in the system is as follows:

[0140]

[0141] in Characterizes the type of workpiece. Characterizing various processing paths for this type of workpiece. and This indicates the start and end of the machining process for this type of workpiece. express Type of workpiece in the first The first step in the process Processing path.

[0142] In the context of Petri nets, the starting library is used. and end of storage To characterize and The activity library represents the workpiece flow relationship along the processing path, and the resource library represents the resource release and use of the machines and transport robots required along the processing path, represented as:

[0143] ,

[0144] After all processing paths for all types of workpieces are constructed, all process sequence subnets and resource subnets are merged to obtain the final network model.

[0145] To characterize the time cost required for each step in the processing path, in each activity warehouse Additional time range for the meeting , Indicates the first Type of workpiece in the first The first step in the process The processing time required for each processing path is not appended to the starting warehouse, ending warehouse, and resource warehouse.

[0146] Step 2: Activate the scheduling decision module. In this embodiment, a scheduling agent is used to train and deploy under varying machine time conditions based on the initial or current resource configuration scheme. This obtains the maximum completion time set across multiple time samples under the current configuration scheme, and the results are fed to the resource configuration decision module's resource configuration agent. The scheduling agent is used to make sequential decisions regarding the processing and transportation processes in the flexible manufacturing system under given resource configuration conditions. This agent takes the system's operating state based on Petri nets as input, selects scheduling actions from currently executable processing or transportation operations, and continuously optimizes the scheduling strategy through reinforcement learning to obtain a smaller maximum completion time. The resource configuration agent is used to determine the configuration quantity of resources such as processing machines and transportation robots in the flexible manufacturing system under cost constraints. This agent adjusts the resource configuration scheme based on the scheduling results feedback, and gradually learns resource configuration strategies that can improve the overall scheduling performance of the system through reinforcement learning.

[0147] See Figure 10 A Dyna-Q tabular multi-agent reinforcement learning algorithm is used for joint optimization of scheduling and resource allocation. For the constructed time-delay Petri net model, an initial resource allocation scheme is given, and the number of processing machines and transport robots in the resource allocation scheme are represented on the resource pool in the time-delay Petri net, i.e., on the corresponding resource pool. The corresponding number of tokens are added to it.

[0148] The training and deployment of the scheduling agent begins based on the current configuration scheme. The final deployment result of the scheduling agent is a set of independent and identically distributed scheduling results (maximum completion time). These results are then input back to the resource allocation agent for sample evaluation, training, and deployment, thereby obtaining a new resource allocation scheme. This process continues iteratively until the joint optimization of scheduling and resource allocation in the flexible manufacturing system is completed, providing a reasonable and excellent resource allocation scheme and the scheduling results under that scheme.

[0149] Based on the final resource allocation and scheduling results, the operations are sorted along a timeline according to the start time of each workpiece and each process, generating an executable production scheduling plan.

[0150] The training and deployment process in a scheduling agent includes the following steps:

[0151] The nominal state of the constructed system is The system sampling status is ,in For Petri net status identifiers, List of nominal remaining time, Used for scheduling state Q-value updates during training. It does not change with the actual remaining processing time, but rather takes the median of the processing time interval. This represents the actual remaining processing time after each action selection following the current machine processing time sampling. It sets the initial state. and target state .

[0152] After obtaining the configuration scheme output by the resource configuration agent, the two-layer loop training of the scheduling agent begins: scenario sampling. The outer loop (outer layer) and the inner loop (sample training layer) are used. The outer loop primarily samples the processing time intervals of each machine, and the total number of outer loop iterations is set. Each cycle uses a specific set of processing time samples. Processing time sampling This includes the processing time of the processing machine and the transportation time of the transport robot. These time samples are then input into the inner loop for training. Each loop operates within the time delay interval of each resource repository. Random sampling is performed to obtain the current Time Sample The sample is then input into the inner loop for training. The inner loop primarily samples the current processing time along with other system information such as the workpiece being processed and the processing steps. It uses an improved Dyna-Q algorithm for training and deployment. After the inner loop finishes, the current scheduling strategy and results are retained, and the next outer layer training is performed. After both inner and outer layer training are completed, the scheduling agent enters the deployment phase after the training phase, where the trained Q_s table is... The second sampling deployment will The results obtained from each sampling deployment are recorded. After the sampling deployment is completed, the final scheduling policy (Q_s table) and... The set of scheduling results corresponding to each time sample under the sub-sampling deployment (the maximum completion time under each time sample) is used as the output of the scheduling agent.

[0153] The deployment results of the scheduling agent are output to the resource allocation agent, mainly to calculate the reward function of the resource allocation agent, thereby optimizing the resource allocation scheme.

[0154] Step 3: After obtaining the output results of the scheduling agent, the resource allocation agent analyzes and evaluates the data, calculates the reward function value, explores the number of machines under the given cost constraint, and provides the new resource allocation scheme to the scheduling agent for training and deployment.

[0155] Step 4: The scheduling agent and the resource allocation agent jointly train through interactive iteration. The resource allocation agent provides the resource configuration, and the scheduling agent trains under the resource configuration and provides the scheduling results for the resource allocation agent to analyze. The two interact and update in a loop until the preset termination condition is reached.

[0156] Step 5: After the multi-agent interactive iterative joint training is completed, the final resource configuration and scheduling results are exported. The operations are sorted on the timeline according to the start time of each workpiece and each process to generate an executable production scheduling plan.

[0157] The improved Dyna-Q algorithm used in the inner training loop includes the following steps:

[0158] Step 001: Set the initial learning rate for scheduling. Discount rate Scheduling replay threshold Scheduling planning steps Total number of training rounds The learning rate decay coefficient of the outermost loop Inner loop learning rate decay coefficient Attenuation ratio Exploration rate return temperature difference Forgetting rate coefficient Number of times the sampling deployment is scheduled Scheduling decay factor The scheduler deadlock penalty is penalty_s, the scheduler target reward is reward_s, and the scheduler policy table Q_s, the scheduler replay list pqueue_s, the scheduler transition list model_s, and the scheduler predecessor list pred_s are initialized.

[0159] Step 002: Initialize the identifier , After sampling the current machine processing time, the actual remaining processing time after each action selection. ;

[0160] Step 003: Based on the current identifier Find the set of enableable transitions under the current Petri net and adopt an improved global exploration rate. A dynamic greedy strategy selects a transition to enable as the current state. The following action Received a new identifier New nominal status and new sampling status The nominal remaining time in the new state The remaining sampling time in the new state is represented by the median of the machine processing interval. Machine processing time based on actual sampling Perform calculations;

[0161] Improved global exploration rate in the Dyna-Q algorithm under a scheduling agent The greedy strategy is designed as follows:

[0162] Set global exploration rate initial value Global exploration rate final value ;

[0163] During the outer training loop of each scenario sampling, the inner exploration rate... Employing an exponential decay strategy, the inner layer... The exploration rate of each round , It is the attenuation factor;

[0164] After completing the inner loop, a warm-up process is required before entering the next outer scenario sampling. This means that whenever entering a new outer scenario... A brief rebound is needed To explore new scenarios ,in This is the temperature difference. This is the exploration rate at the end of the previous inner loop.

[0165] Step 004: Calculate Instant Rewards When the new logo When no enableable transitions are available for action selection, the new state is considered to have entered a deadlock. A negative scheduling deadlock penalty penalty_s; when the new identifier Equal to target identifier At this time The reward is reward_s for a positive scheduling target; in general scenarios, pass The calculation is performed; the general scenario is that after performing an action (launch transition), the new state does not enter a deadlock and is not the target state. In this case, the reward function value needs to be calculated using the current list of actual remaining time.

[0166] Examples of instant rewards in general scenarios Calculation: Assuming the current state, the selected action The corresponding strain change All Pre-Activity Library , , And given that the current actual remaining time for each activity library is 2.4, 1.3, and 1.9 units of time, respectively, then the action in the current state must be completed. Afterwards, the remaining time of all preceding activities in the transition corresponding to the action will be reset to zero, and the largest remaining time will be selected as the immediate reward for the current action (if the new state is not a deadlock state). In this example, 2.4 units of time will be used as the current state to take the action. Instant rewards .

[0167] Step 005: Transfer the state transition sample The state transition is recorded in the scheduling state transition module mode_s, and the set of state predecessors is also recorded in the scheduling predecessor list pred_s. Fill in ;

[0168] Step 006, according to Time difference error Calculated, such as Not deadlocked and Greater than the scheduling replay threshold Then experience Push it into the min-heap scheduling replay list pqueue_s, and then use an improved adaptive decay learning rate. pass Update the Q_s table once;

[0169] Improved adaptive decay learning rate in the Dyna-Q algorithm under a scheduling agent The design is as follows:

[0170] Because we want the model to learn quickly in new scenarios, the learning rate in the Dyna-Q algorithm is... An adaptive decay strategy is adopted when entering a new The effect is brief, followed by a gradual decline:

[0171]

[0172] in r represents the number of iterations in the outer loop. Represents the number of inner loop iterations. This represents the initial learning rate or the learning rate at the end of the previous inner / outer loop. and This represents the decay coefficient of the learning rate in the outer and inner loops;

[0173] Step 007, proceed The scheduling loop performs a virtual replay, popping the highest-priority entry from the min-heap scheduling replay list pqueue_s in each iteration. The priority of the design improvement is to be with Size and current outer are related. The scheduling state transition module model is used to find... Find by referring to the Q_s table The Q value corresponding to the optimal action. Substitute this into the Q-value update formula in step 006 to update the value. Then find the scheduling predecessor set. The set of states, calculating the state of all predecessor states. Greater than the scheduling replay threshold Entries Similarly, push the data into the min-heap scheduling replay list pqueue_s. Repeat this process until completion. Virtual replay of the next scheduling cycle;

[0174] The priority design of the replay list in the Dyna-Q algorithm under the scheduling agent is as follows:

[0175] To emphasize the training effect under the new time samples, when considering A decay weight is added along with the error, increasing the likelihood of new experience entries being replayed while still allowing older ones to be replayed. Each experience entry is recorded with its outermost number, which is used to calculate its decay weight. :

[0176]

[0177] in, This is the forgetting rate coefficient. and This refers to the current outer layer number and the outer layer number corresponding to this experience entry.

[0178] The priority of each empirical item is calculated using attenuation weights. ,in As a decay ratio, the highest priority experience entry will be placed at the top of the min-heap scheduling replay list pqueue_s;

[0179] Step 008: Update the nominal state and the sampled state, then return to step 003 to continue the loop until... Reaching the target state Alternatively, if a deadlock occurs, end the current training round and return to step 002 to begin a new training round. The total number of training rounds has been reached. The outer loop training ends at the specified time;

[0180] Step 009: When all outer loops are scheduled After training, the trained Q_s table is processed. The secondary scheduling sampling deployment follows the same sampling process as the time sample sampling process in the outer sampling layer. From the initial state... Start by finding the transition with the largest Q value in the current state as the action, and record the reward for each action. And it accumulates until the target state is reached. (Rewards for reaching the target state) No need to add positive scheduling target reward It's important to note that the Q_s table is frozen and not updated during the sampling deployment process. The final total reward... The maximum completion time (makespan) obtained by scheduling the agent is the action sequence, which is the machine processing sequence. The results obtained from the next scheduling sampling deployment are recorded. After the sampling deployment is completed, the set of scheduling results (deployment results) and the final scheduling policy (Q_s table) are used as the output of the scheduling agent.

[0181] Here is a brief explanation of the sampling deployment:

[0182] After completing the outer sampling and inner training loops, the training phase of the scheduling agent ends, and the deployment phase begins. The goal of the deployment phase is to obtain the scheduling results under the current resource configuration scheme. Since the machine processing time and the transport robot's transport time in the current system are given as time interval information, it is necessary to provide the scheduling results of multiple independent and identically distributed time sampling samples in order to comprehensively represent the scheduling information under the current resource configuration scheme, thereby better enabling subsequent resource allocation.

[0183] Specifically, the training and deployment process of the resource allocation agent includes the following steps:

[0184] Step 101: Initialize the configuration strategy table Q_a, the replay list pqueue_a, the transition list model_a, and the predecessor list pred_a, and set the learning rate. Configure discount rate Configure playback threshold Configure the number of planned steps Configure the total number of training rounds Initial and final exploration rates , Cost penalty coefficient Cost budget ceiling Configure attenuation factor Configure deadlock penalties Penalties for overspending Uncertainty penalty factor Configure the number of deployments ;

[0185] Step 102: Enter the loop iteration and set the current state. For initial resource configuration ,state Set as a resource configuration vector It represents the number of various processing machines and transport robots;

[0186] Step 103: Obtain the current resource configuration by solving the scheduling decision module. The following is a set of scheduling results ,in This represents the number of scheduling results in the set, i.e., the number of results obtained by the scheduling agent after sampling and deployment;

[0187] Step 104: For the current state List the legal action set The actions Characterized as the first The number of machines can be increased or decreased by one unit, based on the upper limit of the cost budget. Restricted action set .according to Greedy strategy selects action The first step in the training process exploration rate in each round Execute actions Acquire new state ;

[0188] Step 105: Calculate the evaluation indicators It represents the current resource allocation scheme. Select action The size and stability of the subsequent scheduling results. Calculation of reward payout:

[0189]

[0190] in This represents the total cost under the new resource allocation plan. If the new resource allocation plan is unreasonable and leads to budget overruns, a penalty will be imposed. Then revert the new state to the original state. And return to step 104;

[0191] Calculate evaluation indicators Includes the following steps:

[0192] Step 1051: Based on resource allocation The following is a set of scheduling results Calculate its sample index:

[0193] Sample mean:

[0194]

[0195] Sample standard deviation:

[0196]

[0197] Step 1052: Construct a comprehensive evaluation index for the scheduling result set. :

[0198]

[0199] in, Controlling the punishment of uncertainty;

[0200] Step 106: Record the state transition relationships using the configuration transition list model_a and the configuration predecessor list pred_a;

[0201] Step 107: Update Q value ,

[0202] like error Greater than the configured replay threshold Then Push the configuration replay list pqueue_a into the min-heap;

[0203] Step 108, proceed In each loop replay, the highest priority entry in the min-heap configuration replay list pqueue_a is popped. And find the configuration state transition module model_a Find the table Q_a The Q value corresponding to the optimal action. Substitute this into the Q-value update formula in step 107 to update the value. Then find the front-drive configuration based on the set of configurations. The set of states is used to calculate the time limit (td) of all predecessor states, which is greater than the replay threshold. Entries Push the configuration replay list pqueue_a into the min-heap. Repeat this process until completion. This configuration allows for virtual loop replay;

[0204] Step 109: Update Status Add one training round, return to step 103 and continue iterating until the set total number of training rounds is reached. Training is over;

[0205] Step 110: Enter the deployment phase of the resource allocation agent, based on the trained strategy. Starting from the initial configuration, a final configuration is obtained by following a greedy strategy;

[0206] The training and deployment of the resource allocation agent have now been completed. Based on the final resource allocation c*, the scheduling decision module is solved again. During this solution process, the warehouse delay is fixed from the interval to the median of the interval to obtain the scheduling result corresponding to the final configuration.

[0207] At the same time, select the first Add the configuration with the highest Q value to the candidate configuration set (if there is overlap, select only one);

[0208] Step 1011: Perform scheduling and solution on each candidate configuration, and select the configuration with the smallest maximum completion time that meets the cost budget requirements as the final configuration.

[0209] The following description, in conjunction with the accompanying drawings, further illustrates this embodiment:

[0210] See Figure 6 The joint optimization method for FMS scheduling and resource allocation includes the following steps:

[0211] Based on actual production requirements, the S-value of the Petri network-based storage delay is used. 3 PR network models the flexible manufacturing system; the warehouse delay S of the flexible manufacturing system 3 The PR net model includes a set of types and numbers of components to be processed, a set of processing machine resources, a set of processing paths, and a set of machine processing times. The dynamic behavior of the system is defined by initial identifiers and state transition formulas.

[0212] Taking into account variations in machine processing time, an improved Dyna-Q algorithm is used for training to obtain the maximum completion time under the current resource configuration.

[0213] The results obtained by the scheduling decision module are used as the reward for each action. An improved Dyna-Q algorithm is used for training to continuously optimize the resource allocation scheme and obtain the optimized resource allocation scheme.

[0214] The final scheduling result is generated based on the optimized resource allocation scheme and machine processing time.

[0215] Specifically as follows:

[0216] Step 1111: Construct a time-delay Petri net model based on a flexible manufacturing system example;

[0217] Depend on Figure 8 It can be seen that there are two types of workpieces in the current system. , Processing is required, and the processing sequence for each workpiece is given:

[0218] For workpiece one In general, it takes five processes (including transportation and processing) to complete the processing of this workpiece, and the order of these processes cannot be changed. Workpiece 1 It needs to pass through the No. 1 transport robot first. After the corresponding transportation process, on the No. 1 processing machine Or the No. 2 processing machine The corresponding processing steps are completed on the first stage, and then the first transport robot passes through it. After completing the corresponding transportation process, at processing machine number three... After completing the corresponding processing steps, it passes through the No. 2 transport robot. After completing the corresponding transportation process, the workpiece is completed. All processing work.

[0219] Similarly, for workpiece two For example, it needs to first pass through the No. 2 transport robot. After the corresponding transportation process, it is processed on machine number four. The corresponding processing steps are completed on the first stage, and then the first transport robot passes through it. After completing the corresponding transportation process, at processing machine number two... After completing the corresponding processing steps, it passes through the No. 1 transport robot. After completing the corresponding transportation process, complete workpiece two. All processing work.

[0220] Figure 9 for , The processing diagrams for the two types of workpieces clearly show that there are multiple processing paths, multiple processing elements, non-unique numbers of transport trolleys and processing machines, resource sharing, and resource occupation conflicts during the processing of the two types of workpieces. Figure 9 middle The outer circle represents the storage area, and the inner circle represents the storage area. Indicates Token, It indicates change.

[0221] Based on current examples of flexible manufacturing systems, transport robot , Use resource library , This indicates that the processing machine , , , Use resource library , , , The number of its transport robots and processing machines is represented by tokens in the resource repository. (Using the starting repository) , Characterization , Two types of workpiece start processing buffer zones, with end storage. , Characterization , Two types of workpiece end-of-process buffer zones. The workpiece's state during the completion process is represented by an active library. To characterize, such as for workpiece one When it performs its first process (the No. 1 transport robot) Transportation) Corresponding activity warehouse If there is no Token in the workpiece, then the workpiece is considered to be one. The first process has not yet been carried out and the workpiece is still within the initial processing buffer. If the active warehouse contains a token and the token currently has remaining processing (transportation) time, then the workpiece is considered to be in the initial processing buffer. In the process of executing the procedure (Robot No. 1 transport vehicle) Workpiece is being transported If the remaining processing (transportation) time of the workpiece reaches zero, it means that the workpiece is ready for transport. The first process has been completed and is awaiting the second process (processing machine number one). Or the No. 2 processing machine The execution of processing steps.

[0222] In the starting, ending, and resource warehouses, the number of tokens in each warehouse represents the number of processing elements contained in the transport trolleys or processing machines in the current process. Its size is constrained by the number of tokens in the corresponding resource warehouse (representing the number of transport trolleys or processing machines, given by resource allocation). Each token in a warehouse has a remaining processing (transport) time, which represents the processing (transport) time required for the processing (transport) element to complete the current processing (transport) process.

[0223] Because the machining (cart transport) time is not constant in actual production and often fluctuates within a certain range (randomness or affected by workpiece / tool / environment), a time delay interval is added to each resource library to characterize the time taken for machining (cart transport) in the actual machining (transport) process. For example... Figure 7 In this example, the time delay interval satisfies the mapping:

[0224]

[0225]

[0226]

[0227]

[0228] The cost information for each resource is mapped as follows:

[0229]

[0230]

[0231] Through changes The enabling transitions are used to realize the transfer of tokens between warehouses and places, which represents the selection of a specific processing machine (cart) to perform the processing (transportation) operation during the actual processing (transportation) process. In this example, the directed arc weights between warehouses and transitions are all 1, representing that a processing machine (cart) on each operation of various processed workpieces can only process (transport) one component at a time. The enabling transitions corresponding to the processing machines (carts) are shown in the table below.

[0232]

[0233] The machining path of a workpiece is characterized by locations and transitions:

[0234] Workpiece 1:

[0235] or

[0236] Workpiece 2:

[0237] After analyzing and modeling the production requirements of the flexible manufacturing system instance, multi-agent reinforcement learning training and deployment are carried out based on the modeled time-delay Petri net model under the current instance, and finally a set of resource allocation schemes and scheduling results under the schemes are obtained.

[0238] Multi-agent reinforcement learning mainly consists of two parts: a scheduling agent (scheduling solver) and a resource allocation agent. The scheduling agent and the resource allocation agent jointly train through interactive iteration. The resource allocation agent provides the resource allocation, and the scheduling agent trains under the resource allocation and provides the scheduling results for the resource allocation agent to analyze. The two interact and update cyclically until a preset termination condition is reached, and the final resource allocation and scheduling results are derived.

[0239] Step 1112: Use the Dyna-Q tabular multi-agent reinforcement learning algorithm to jointly optimize scheduling and resource allocation;

[0240] For a pre-constructed Petri net model of place delay, given a set of initial resource configuration parameters... Initial identifier The target is identified as .

[0241] First, we will proceed with the training and deployment of the resource allocation agent:

[0242] 1) Initialize configuration strategy Table _a, Configuration Replay List Configure transfer list and front-wheel drive configuration list Configure learning rate Set the discount rate to 0.05. The value is 0.98, and the total number of training rounds is configured. The initial and final exploration rates are 2000. , The cost penalty coefficients are 0.8 and 0.05. The upper limit of the cost budget is 100. Configure the attenuation factor to 25000. Set the value to 0.995 and configure deadlock penalty. The penalty for exceeding the budget is 1000. The uncertainty penalty factor is 5000. Set the number of deployments to 10. It is 7;

[0243] 2) Use the current state as the initial state of the resource allocation agent. ,according to Greedy strategy selects one action And execute, assuming that processing machine number one has been selected. Performing the increment action will obtain the new resource configuration. And corresponding to the new state Calculate the new state The corresponding cost: Therefore, it meets the resource cost constraints and can be used for subsequent scheduling solutions (such as a certain resource allocation). The corresponding cost is At this time, resource allocation should be maintained. Without changing the reward function and without performing subsequent scheduling, let it remain unchanged. (The status is not updated).

[0244] 3) Provide a new initial identifier for the Petri net based on the new resource configuration. and new target identifiers Based on this, the scheduling agent under this resource configuration is trained and deployed:

[0245] 3.1) Set the total number of outer loop iterations The total number of inner scheduling loops is 10. The initial learning rate is set to 10000. Set to 0.08, scheduling discount rate The threshold for scheduling replay is 0.98. The number of scheduling planning steps is 0.01. The learning rate decay coefficient of the outer loop is 20. The learning rate decay coefficient of the inner loop is 0.01. The exploration rate recovery temperature difference is 0.001. The forgetting rate coefficient is 0.2. The number of times the sampling deployment is 0.3 is scheduled. It is 7;

[0246] 3.2) Enter the outer loop and perform time sampling (with a precision of 0.1) to obtain the current time sample of the outer loop:

[0247]

[0248]

[0249]

[0250]

[0251] 3.3) Enter the inner loop and determine the current state flag. by Find the set of enableable transitions (action space): To improve Dynamic Greedy Strategy Selection As the current state The following action Obtain a new status identifier ,in , ;

[0252] 3.4) Calculate instant rewards Due to the change in the chosen action Pre-position warehouse ( If there is no delay, then ;

[0253] 3.5) State transition samples Record the changes in the scheduling transfer module model_s and the list of scheduling predecessors pred_s;

[0254] 3.6) Calculation error: Therefore, experience Instead of pushing the data into the min-heap, replay the data and perform a Q-table update. Since the current experience cannot demonstrate the complete scheduling and training process, it is assumed here that an experience was found in a certain loop. Play back the video; in this experience After priority calculation, this experience has the highest priority and is therefore located at the top of the min-heap replay list. During the replay process, the Q-value is updated again, and the experiences of all its predecessor states are found and their values ​​are calculated. After adding to the replay list, continue replaying until the maximum number of replays is reached;

[0255] 3.7) Update Status The identifier , , ;

[0256] 3.8) Repeat steps 3.3) to 3.7) until the state reaches the target state or a deadlock state, then the inner loop ends. After completing the specified number of inner loops, the next outer sampling loop begins. 3.2) Repeat this process until the outer loop count is reached, then the scheduling agent training phase ends.

[0257] 3.9) After the training phase is completed, the current resource allocation The following sampling deployment results :

[0258]

[0259]

[0260] 4) Calculate the new resource configuration based on the sampling deployment results. Evaluation indicators And calculate the resource allocation scheme. Select action After obtaining the state Rewards and benefits :

[0261]

[0262]

[0263]

[0264]

[0265] 5) Apply state transition experience Record the configuration transfer module model_a and the configuration predecessor set list pred_a;

[0266] 6) Update the Q_a table once based on current experience, which is basically similar to 3.6) in the scheduling agent, and will not be repeated here;

[0267] 7) Perform experience replay and update the Q_a table again, which is basically similar to 3.7) in the scheduling agent, so it will not be described in detail here;

[0268] 8) Update status Add one training round, return to step 3) and continue the loop iteration until the total number of training rounds is reached as configured. The training phase of the resource allocation agent has ended;

[0269] 9) Based on the trained strategy Starting from the initial configuration, a final configuration scheme is obtained by following a greedy strategy: The cost of 22920 meets the cost ceiling. Using this configuration scheme, the scheduling agent is used to solve the problem, and the scheduling result is obtained:

[0270] Transition sequence:

[0271]

[0272]

[0273] Maximum completion time: 28

[0274] Step 1113: Export the final resource configuration scheme and scheduling scheme, and give the machine processing time based on the current scheduling scheme;

[0275] The trigger times for each transition, obtained from the transition sequence based on the scheduling results, are shown in the table below:

[0276]

[0277] The corresponding Gantt chart of scheduling results in a real flexible manufacturing system is as follows: Figure 11 As shown, 1-1, 1-2, 1-3 and 2-1, 2-2, 2-3 respectively represent as The first, second, and third workpieces and The first, second, and third workpieces.

[0278] See Figure 5 The FMS scheduling and resource allocation joint optimization system includes:

[0279] The model building module uses the S-type delay of the Petri net with a storage area based on actual production requirements. 3 PR network models the flexible manufacturing system; the warehouse delay S of the flexible manufacturing system 3 The PR net model includes a set of types and numbers of components to be processed, a set of processing machine resources, a set of processing paths, and a set of machine processing times. The dynamic behavior of the system is defined by initial identifiers and state transition formulas.

[0280] The scheduling agent module, taking into account the changes in machine processing time, is trained using an improved Dyna-Q algorithm to obtain the maximum completion time under the current resource configuration.

[0281] The resource allocation agent module uses the results solved by the scheduling decision module as the reward for each action, and is trained using an improved Dyna-Q algorithm to continuously optimize the resource allocation scheme and obtain the optimized resource allocation scheme.

[0282] The final scheduling result generation module is used to generate the final scheduling result based on the optimized resource configuration scheme and machine processing time.

[0283] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0284] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0285] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0286] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0287] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A joint optimization method for FMS scheduling and resource allocation, characterized in that, Includes the following steps: Based on the actual production relationship of the Flexible Manufacturing System (FMS), an S-type system with warehouse delay is adopted. 3 The PR network constructs the Petri net model of the FMS to obtain the S with place delay. 3 PR network model, the S with library delay 3 The PR net model represents the set of workpieces, process sequence, resource set, and time characteristics of each process in FMS. The resource set includes processing machines and transport robots. The scheduling decision module is activated, taking into account the S-type delay of the warehouse. 3 The PR network model, based on the initial or current resource allocation scheme, executes an improved Dyna-Q algorithm under varying machine time conditions to obtain the maximum completion time set under multiple time samples. The scheduling decision module includes scheduling strategy learning and scheduling strategy application. The scheduling strategy learning includes an outer loop and an inner loop. The outer loop randomly samples the processing time interval and transportation time interval of each process to obtain time samples. The inner loop, based on the time samples, completes the scheduling strategy learning through the improved Dyna-Q algorithm, and then sends the scheduling decision results corresponding to each set of time samples to the resource allocation decision module through scheduling strategy application. The resource allocation decision module is activated to analyze and evaluate the scheduling decision results set under multiple time samples, calculate the reward function value, explore the number of machines under cost constraints, generate a new resource allocation scheme and feed it back to the scheduling decision module. Based on the new resource allocation scheme, the scheduling strategy learning and application of the scheduling decision module are repeated to obtain a new set of scheduling decision results under multiple time samples; based on the new set of scheduling decision results under multiple time samples, the resource allocation decision module is repeatedly started, and joint optimization iteration is performed until the preset termination condition is met; the goal of the joint optimization iteration is to minimize the maximum system completion time under the premise of satisfying resource and cost constraints. After the joint optimization iteration is completed, the final resource allocation and scheduling results are output. The time axis is sorted according to the start time of each workpiece and each process to generate an executable production scheduling plan. The improved Dyna-Q algorithm includes the following steps: Step 1: Initialize scheduling parameters, scheduling policy table, scheduling replay list, scheduling transfer list, and scheduling predecessor list; Step 2: Initialize the identifier to obtain the current identifier, and initialize the nominal remaining time and the actual remaining processing time after the action selection; Step 3: Based on the current identifier obtained in Step 2, determine the set of enabling transitions under the Petri net, and adopt an improved global exploration rate. The greedy strategy selects the transition enable as the action in the current state, obtains a new identifier and the corresponding nominal state, sampled state and new state, the new state is the set of the nominal state and sampled state, and generates state transition samples. Step 4: Determine the state type of the new identifier obtained in Step 3, and calculate the immediate reward based on the state type; the state types include deadlock state, target identifier state, and general state, and different state types correspond to different immediate reward calculation methods; Step 5: Record the state transition sample generated in Step 3 into the scheduling transition list, and at the same time record the current state as the predecessor state of the new state obtained in Step 3, and store it in the scheduling predecessor list. Step 6: Reuse the immediate reward from Step 4 and the new state from Step 3 to calculate the timing difference error. If the new state is not a deadlock state and the timing difference error meets the preset conditions, store the corresponding experience in the scheduling replay list, and then use the improved adaptive decay learning rate to update the scheduling strategy table. Step 7: Perform virtual replay a preset number of times. Each time, extract the highest priority experience from the scheduling replay list. Based on the scheduling transfer list constructed in Step 5, determine the new state corresponding to the highest priority experience. Update the policy value corresponding to the highest priority experience in conjunction with the scheduling policy table. At the same time, process the timing difference error of the predecessor state based on the scheduling predecessor list in Step 5, and store the experience that meets the conditions into the scheduling replay list. Step 8: After updating the nominal state and sampled state obtained in Step 3, return to Step 3 and continue the loop until the new state is the target state or deadlock state to end this round of optimization; repeat the above optimization process until the preset total number of optimization rounds is reached to complete the outer loop optimization; Step 9: After completing the training of all outer loops in Step 8, the trained scheduling strategy table is sampled and deployed a preset number of times; starting from the initial state, the action with the optimal strategy value is selected for execution, the reward of each action is recorded and accumulated until the target state is reached, the scheduling strategy table is kept frozen during the sampling and deployment process, and finally the sampling and deployment result set and the scheduling strategy table are output.

2. The joint optimization method for FMS scheduling and resource allocation according to claim 1, characterized in that, The specific process of learning the scheduling strategy in the scheduling decision module is as follows: Construct the nominal system state and the sampled system state; both the nominal system state and the sampled system state include Petri net state identifiers and corresponding remaining processing time. The nominal system state corresponds to the nominal remaining time, and the sampled system state corresponds to the actual remaining processing time after sampling the current machine processing time and each action selection. At the same time, set the initial state and the target state of the system. Obtain the initial or current resource configuration scheme output by the resource configuration decision module, and perform outer loop training and inner loop training of the scheduling decision module. The outer loop training includes sampling the processing time interval of each machine, setting the total number of outer loops, using a set of processing time sampling samples for each outer loop, and inputting the processing time sampling samples into the inner loop for sample training. The inner loop training includes sampling samples and system information based on the current processing time, learning and applying the scheduling strategy of the samples through the improved Dyna-Q algorithm, obtaining the scheduling strategy and result of the scheduling decision module, and retaining the currently trained scheduling strategy and result of the scheduling decision module after the inner loop ends and performing the next outer loop training. The specific process of applying the scheduling strategy in the scheduling decision module is as follows: After the outer loop training and inner loop training of the scheduling decision module are completed, the Q_s table of the trained scheduling decision module is sampled and deployed multiple times, and the results obtained from the multiple sampling deployments are recorded. After the sampling deployment is completed, the final Q_s table and the set of maximum completion times for each time sample are obtained.

3. The joint optimization method for FMS scheduling and resource allocation according to claim 1, characterized in that, The improved global exploration rate Greedy strategy, specifically: Set global exploration rate initial value Global exploration rate final value ; During the outer training loop of each scenario sampling, the internal global exploration rate... Employing an exponential decay strategy, the inner layer... The exploration rate of each round , It is the attenuation factor; After completing the inner loop, a temperature recovery process is performed when entering the next outer loop scenario sampling. This process is repeated whenever a new scenario is entered. The cycle needs a brief rebound. Exploring new scenarios ,in This is the temperature difference. This is the exploration rate at the end of the previous inner loop; The adaptive decay learning rate is shown in the following formula: in, This represents the adaptive decay learning rate. r represents the number of iterations in the outer loop. Represents the number of inner loop iterations. This represents the initial learning rate or the learning rate at the end of the previous inner / outer loop. and This represents the decay coefficient of the learning rate in the outer and inner loops.

4. The joint optimization method for FMS scheduling and resource allocation according to claim 1, characterized in that, The priority of the playback list in the improved Dyna-Q algorithm takes into account timing difference errors. Simultaneously, a decay weight is added, and the priority of each experience entry is calculated using the decay weight. The experience entry with the highest priority is placed at the top of the min-heap replay list. The decay weight is shown in the following formula: in, For decay weights, This is the forgetting rate coefficient. and This refers to the current outer layer number and the outer layer number corresponding to the experience entry.

5. The joint optimization method for FMS scheduling and resource allocation according to claim 1, characterized in that, The resource allocation learning process of the resource allocation decision module includes the following steps: Step 01: Initialize the resource configuration strategy table, configuration replay list, configuration state transition list, and configuration predecessor list, and set the training parameters to obtain the initial resource configuration; Step 02: After initialization is completed, enter the loop iteration, and take the initial resource configuration as the initial state. The initial state represents the quantity configuration of various processing equipment and transportation equipment. Step 03: Based on the resource configuration corresponding to the current initial state, solve the scheduling decision module to obtain the set of scheduling results under the current resource configuration; Step 04: After obtaining the scheduling result set, determine the set of legal actions for the current state. The set of legal actions is subject to cost budget constraints. Select and execute actions based on a greedy strategy to obtain a new state. Step 05: After obtaining the new state, calculate the evaluation index and reward corresponding to the new state; if the resource allocation corresponding to the new state exceeds the cost budget, apply a penalty and revert the state to the original state, then return to step 4 to select an action again; Step 06: After completing the reward calculation and state rollback processing, record the state transition relationships by configuring the state transition list and the predecessor list; Step 07: After recording the state transition relationship, update the Q value; if the timing difference error exceeds the preset threshold, push the state action information into the configuration playback list; Step 08: After completing the Q-value update and pushing the configuration replay list entries, perform a preset number of loop replays. Each time, the highest priority entry is popped from the configuration replay list. The subsequent state is determined based on the configuration state transition list. The Q-value corresponding to the optimal action in the subsequent state is found by combining the configuration strategy table, and the Q-value of the current entry is updated. At the same time, the set of predecessor states of the current state is determined based on the configuration predecessor list. The timing difference error of each predecessor state is calculated. The predecessor state information that meets the threshold condition is pushed into the configuration replay list until the preset number of loop replays is completed. Step 09: After completing the preset number of loop replays, update the current state to the new state, accumulate the training rounds, and return to Step 3 to continue iterating until the preset total number of optimization rounds is reached, and the optimization ends; Step 010: After optimization, the deployment phase begins. Based on the optimized strategy, the final resource configuration is obtained from the initial configuration using a greedy strategy. The scheduling decision module is then executed again based on the final resource configuration to obtain the final scheduling result.

6. The joint optimization method for FMS scheduling and resource allocation according to claim 5, characterized in that, The calculation process of the evaluation indicators in the resource allocation decision module includes: The sample mean and sample standard deviation are calculated based on the scheduling result set. Based on the sample mean and sample standard deviation, a comprehensive evaluation index is constructed using the comprehensive evaluation index formula. The calculation of the sample mean and sample standard deviation based on the scheduling result set is shown in the following formula: in, The sample mean. The standard deviation of the sample is 1. For the number of sampling deployments, For current resource allocation The next The scheduling results for each sample; The formula for the comprehensive evaluation index is as follows: in, As a comprehensive evaluation indicator, As an uncertainty penalty factor, ; When outputting the final resource configuration and scheduling results, a set of candidate configurations with the top Q values ​​after training is selected, and each configuration is re-scheduled and solved. The configuration that meets the cost budget and has the shortest maximum completion time is selected as the final resource configuration.

7. An FMS scheduling and resource allocation joint optimization system, characterized in that, The FMS scheduling and resource allocation joint optimization method according to any one of claims 1-6 includes: The model building module is used for the actual production relationships based on the Flexible Manufacturing System (FMS), employing S with warehouse delay. 3 The PR network constructs the Petri net model of the FMS to obtain the S with place delay. 3 PR network model, the S with library delay 3 The PR net model represents the set of workpieces, process sequence, resource set, and time characteristics of each process in FMS. The resource set includes processing machines and transport robots. The scheduling decision module is used to combine the S with the warehouse delay. 3 The PR network model, based on the initial or current resource allocation scheme, executes an improved Dyna-Q algorithm under varying machine time conditions to obtain the maximum completion time set under multiple time samples. The scheduling decision module includes scheduling strategy learning and scheduling strategy application. The scheduling strategy learning includes an outer loop and an inner loop. The outer loop randomly samples the processing time interval and transportation time interval of each process to obtain time samples. The inner loop, based on the time samples, completes the scheduling strategy learning through the improved Dyna-Q algorithm, and then sends the scheduling decision results corresponding to each set of time samples to the resource allocation decision module through scheduling strategy application. The resource allocation decision module is used to analyze and evaluate the set of scheduling decision results under multiple time samples, calculate the reward function value, explore the number of machines under cost constraints, generate new resource allocation schemes and feed them back to the scheduling decision module. The scheduling and resource allocation joint optimization module is used to repeatedly learn and apply the scheduling strategy of the scheduling decision module based on the new resource allocation scheme, and obtain a new set of scheduling decision results under multiple time samples; based on the new set of scheduling decision results under multiple time samples, the resource allocation decision module is repeatedly started, and the joint optimization iteration is carried out until the preset termination condition is met; the goal of the joint optimization iteration is to minimize the maximum system completion time under the premise of satisfying resource and cost constraints. The production scheduling scheme generation module is used to output the final resource configuration and scheduling results after the joint optimization iteration is completed. It sorts the time axis according to the start time of each workpiece and each process, and generates an executable production scheduling scheme.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the FMS scheduling and resource allocation joint optimization method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the FMS scheduling and resource allocation joint optimization method as described in any one of claims 1-6.

Images