FMS scheduling and resource configuration joint optimization method, system, equipment and medium

Abstract

The invention discloses an FMS scheduling and resource configuration joint optimization method, system, device and medium, and belongs to the field of manufacturing system optimization, and the method comprises the steps: employing an S3PR network with library time delay to construct a Petri network model of an FMS, and representing workpieces and processes in the FMS; the scheduling decision module executes an improved Dyna-Q algorithm in a machine time change environment based on resource configuration to obtain a maximum completion time set of multiple groups of time samples; the resource configuration decision-making module analyzes the set, calculates a reward function value, explores machine number configuration under cost constraint, generates a new scheme and feeds back the new scheme to the scheduling decision-making module; the method and the system are subjected to interactive iteration joint optimization to meet resource and cost constraints and minimize the maximum completion time of the system, after a preset termination condition is achieved, a resource configuration and scheduling result is output, an executable scheme is generated according to work procedure starting time sorting, and the quality and efficiency of scheduling and resource configuration are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of manufacturing system optimization, and particularly relates to a FMS scheduling and resource allocation joint optimization method, system, device and medium. BACKGROUND

[0002] FMS (Flexible Manufacturing System) is a key means to cope with market diversification and order uncertainty. FMS contains multiple processes, multiple workpieces and multiple types of equipment, and resources are limited and have concurrent and synchronous relationships, so scheduling and resource allocation are the core problems affecting system performance. Petri net, as a modeling tool for discrete event systems, can clearly express concurrency, resource competition and synchronization constraints, and is widely used to describe and solve scheduling problems of manufacturing systems. Traditional optimization methods (heuristic, genetic algorithm, ant colony, etc.) can achieve good solutions on small and medium-sized problems, but when the state space explodes or the time / process parameters change dynamically, they often fall into local optimum or are difficult to maintain robustness.

[0003] In recent years, reinforcement learning has been introduced into manufacturing scheduling research. The method based on Q-learning (action value function learning) gradually improves the strategy by interacting with the environment, and has the advantages of not needing strong prior models and having online adaptive ability. However, in actual large-scale FMS, reinforcement learning faces several challenges: first, when faced with large-scale manufacturing systems, a large number of deadlock states will affect the agent's exploration of the optimal scheduling path, thereby reducing the quality of scheduling solution; second, in actual manufacturing scheduling problems, resource allocation will also greatly affect the scheduling results of the manufacturing system, and evaluating a resource allocation requires the consumption of complete scheduling solution resources, thereby seriously affecting the overall solution efficiency; third, machine processing time in actual manufacturing systems is not constant, but often fluctuates within a certain range (randomness or affected by workpieces / tools / environment), so that the strategy trained from the deterministic model may perform poorly in the actual scene, and traditional reinforcement learning and planning based on a single deterministic time model cannot effectively cope with this time uncertainty.

[0004] To this end, a robust training and evaluation mechanism for uncertainty needs to be introduced in the method. The present application proposes to describe the machining time with a time interval, and to explicitly incorporate uncertainty into the training and evaluation process through scenario sampling: at the scheduling end, the internal model of Dyna-Q (Dynamic Q-learning) is changed from single-value mapping to sample pool model, and the non-deadlock samples of different sampling scenarios are played back in the priority playback and planning stage to estimate the expected return of each state corresponding action; at the resource allocation end, multiple sampling evaluations are taken for each candidate configuration and the statistics are cached to reduce the expensive repeated scheduling calls and obtain robust configuration comparison indicators. In addition, the double-layer loop of outer scenario sampling and inner training can cover more time scenarios and improve the generalization ability of the strategy to time fluctuations.

[0005] The above improvements aim to solve three types of deficiencies: first, break the optimization mode of separation of scheduling and resource allocation, and build a joint training framework of scheduling and resource allocation of flexible manufacturing system considering the uncertainty of machining or transportation time in the scheduling process; second, improve the scheduling solution quality of large-scale flexible manufacturing system with a large number of deadlock states by reinforcement learning; third, significantly reduce the computing resources consumed in the resource allocation process, and provide resource allocation schemes that take into account both the expected maximum completion time and risk assessment under budget constraints. Therefore, in the real flexible manufacturing scene with time fluctuations, there is an urgent need for a reinforcement learning joint optimization method that combines Petri net modeling, sample pool planning, outer scenario sampling, and multi-sampling configuration evaluation to improve the overall performance and decision reliability of the system. SUMMARY

[0006] The present application aims to overcome the complex scheduling problem of large-scale flexible manufacturing systems and proposes a FMS scheduling and resource allocation joint optimization method, system, device and medium.

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions: In a first aspect, the present application provides a FMS scheduling and resource allocation joint optimization method, comprising the following steps: Based on the actual production relationship of the flexible manufacturing system FMS, the S 3 PR network is used to build a Petri network model of FMS, and a S 3 PR network model, a S 3 The PR network model represents the workpiece set, process sequence, resource set and time characteristics of each process in the FMS, and the resource set includes machining machines and transportation robot machines; Start the scheduling decision module, combine the S 3The PR net model, based on an initial or current resource configuration scheme, executes an improved Dyna-Q algorithm in a machine time-varying environment to obtain a maximum completion time set under multiple groups of time samples; the scheduling decision module includes scheduling strategy learning and scheduling strategy application; the scheduling strategy learning includes an outer loop and an inner loop, the outer loop randomly samples the processing time interval and the transportation time interval of each process to obtain time samples; the inner loop completes scheduling strategy learning through the improved Dyna-Q algorithm based on the time samples, and then delivers the scheduling decision results corresponding to each group of time samples to the resource configuration decision module through the scheduling strategy application; The resource configuration decision module is started to analyze and evaluate the scheduling decision result set under multiple groups of time samples, and to calculate the reward function value, to explore the number of machines under the cost constraint, to generate a new resource configuration scheme and to feed back the scheduling decision module; The scheduling strategy learning and the scheduling strategy application of the scheduling decision module are repeated based on the new resource configuration scheme to obtain a new scheduling decision result set under multiple groups of time samples; the resource configuration decision module is repeatedly started based on the new scheduling decision result set under multiple groups of time samples, and joint optimization iteration is performed until a preset termination condition is met; the target of the joint optimization iteration is to minimize the system maximum completion time under the premise of meeting the resource and cost constraints; After the joint optimization iteration ends, the final resource configuration and scheduling result are output, the time axis is sorted according to the start time of each process of each workpiece, and an executable production scheduling scheme is generated.

[0008] Further, the scheduling strategy learning of the scheduling decision module specifically includes: The nominal state of the system and the sampling state of the system are constructed; the nominal state of the system and the sampling state of the system both include the Petri net state identifier and the corresponding processing remaining time, the nominal state of the system corresponds to the nominal remaining time, and the sampling state of the system corresponds to the actual processing remaining time after sampling and each action selection of the current machine processing time; meanwhile, the initial state and the target state of the system are set; The initial or current resource configuration scheme output by the resource configuration decision module is obtained, and the outer loop training and the inner loop training of the scheduling decision module are performed; the outer loop training includes sampling the processing time interval of each machine, setting the total number of outer loop cycles, using a group of processing time sampling samples for each outer loop, and inputting the processing time sampling samples to the inner loop for sample training; The inner loop training includes scheduling strategy learning and scheduling strategy application of the sample through the improved Dyna-Q algorithm based on the current processing time sampling sample and the system information, obtaining the scheduling strategy and the result of the scheduling decision module, and retaining the scheduling strategy and the result of the trained scheduling decision module obtained at present after the inner loop ends and performing the next outer loop training; The application specific process of the scheduling strategy of the scheduling decision module is as follows: After the outer loop training and the inner loop training of the scheduling decision module are completed, the Q_s table of the trained scheduling decision module is sampled and deployed multiple times, the results obtained by the multiple sampling and deployment are recorded, and the final Q_s table and the maximum completion time set under each time sample are obtained after the sampling and deployment are completed.

[0009] Further, the improved Dyna-Q algorithm comprises the following steps: Step 1: initialize the scheduling parameters, the scheduling strategy table, the scheduling replay list, the scheduling transition list and the scheduling predecessor list; Step 2: initialize the current identifier, initialize the nominal remaining time and the actual processing remaining time after action selection; Step 3: based on the current identifier obtained by the initialization in step 2, determine the set of enabled transitions under the Petri net, and use the improved global exploration rate The greedy strategy selects the transition enablement as the action in the current state, obtains the new identifier and the corresponding nominal state, the sampling state and the new state, the new state is the set of the nominal state and the sampling state, and generates a state transition sample; Step 4: judge the state type according to the new identifier obtained in step 3, and calculate the immediate reward according to the state type; the state type includes the deadlock state, the target identifier state and the general state, and different state types correspond to different immediate reward calculation methods; Step 5: record the state transition sample generated in step 3 to the scheduling transition list, record the current state as the predecessor state of the new state obtained in step 3, and store it in the scheduling predecessor list; Step 6: reuse the immediate reward in step 4 and the new state in step 3 to calculate the timing difference error, if the new state is not a deadlock state and the timing difference error meets the preset condition, store the corresponding experience into the scheduling replay list, and update the scheduling strategy table using the improved adaptive decay learning rate; Step 7: perform virtual replay for a preset number of times, extract the highest priority experience in the scheduling replay list each time, determine the new state corresponding to the highest priority experience based on the scheduling transition list constructed in step 5, and update the strategy value corresponding to the highest priority experience in combination with the scheduling strategy table; at the same time, process the timing difference error of the predecessor state based on the scheduling predecessor list in step 5, and store the experience meeting the condition into the scheduling replay list; Step 8: after the nominal state and the sampling state obtained in step 3 are updated, return to step 3 to continue the loop until the new state is the target state or the deadlock state to end this round of optimization; repeat the above optimization process until the preset total optimization round is reached, and complete the outer loop optimization; Step 9: After all the outer loop training is completed in step 8, the trained scheduling policy table is sampled and deployed for a preset number of times; starting from the initial state, the action with the optimal policy value is selected and executed, the reward of each action is recorded and added to the target state, the scheduling policy table remains frozen during the sampling and deployment process, and the final output is a set of sampling and deployment results and the scheduling policy table.

[0010] Further, the improved global exploration rate The greedy strategy is as follows: The global exploration rate is set The initial value is The final value of the global exploration rate ; During the outer loop training cycle of each scenario sampling, the internal global exploration rate An exponential decay strategy is adopted, and the exploration rate of the internal layer in the th round is , is the decay factor; After completing the internal loop, the next outer loop scenario sampling is entered for re-warming processing. Whenever a new cycle is entered, the exploration rate needs to be temporarily increased to explore new scenarios , wherein is the re-warming difference value, is the exploration rate at the end of the previous internal loop; Adaptive decay learning rate, as shown in the following formula:

[0011] wherein, represents the adaptive decay learning rate, r represents the number of outer loops, represents the number of internal loops, represents the initial learning rate or the learning rate at the end of the previous internal / outer loop, and represent the decay coefficients of the learning rate in the outer and internal loops.

[0012] The priority of the replay list of the improved Dyna-Q algorithm considers the temporal difference error and adds a decay weight, which is used to calculate the priority of each experience item. The experience item with the highest priority is placed at the top of the min-heap replay list. The decay weight is shown in the following formula:

[0013] wherein, is the decay weight, is the forgetting speed coefficient, and the current outer layer number and the outer layer number corresponding to the experience entry.

[0014] Further, the resource configuration learning process of the resource configuration decision module includes the following steps: Step 01: initialize the configuration strategy table, the configuration playback list, the configuration state transition list and the configuration predecessor list of the resource configuration, and set the training parameters to obtain the initial resource configuration; Step 02: after the initialization is completed, enter the loop iteration, and take the initial resource configuration as the initial state, which represents the number configuration of various processing equipment and transportation equipment; Step 03: based on the resource configuration corresponding to the current initial state, solve through the scheduling decision module to obtain a scheduling result set under the current resource configuration; Step 04: after obtaining the scheduling result set, determine a legal action set for the current state, which is limited by the cost budget; select an action based on the greedy strategy and execute it to obtain a new state; Step 05: after obtaining the new state, calculate the evaluation index and reward return corresponding to the new state; if the resource configuration corresponding to the new state exceeds the cost budget, apply a penalty and return the state to the original state, and return to step 4 to select an action again; Step 06: after completing the reward return calculation and state rollback processing, record the state transition relationship through the configuration state transition list and the configuration predecessor list; Step 07: after recording the state transition relationship, update the Q value; if the time difference error exceeds the preset threshold, push the state action information into the configuration playback list; Step 08: after completing the Q value update and pushing the configuration playback list entries, perform a preset number of loop playback, each time pop out the highest priority entry from the configuration playback list, determine the subsequent state based on the configuration state transition list, find the Q value of the optimal action under the subsequent state in combination with the configuration strategy table, and update the Q value of the current entry; at the same time, determine the predecessor state set of the current state according to the configuration predecessor list, calculate the time difference error of each predecessor state, and push the predecessor state information meeting the threshold condition into the configuration playback list until the preset number of loop playback is completed; Step 09: after completing the preset number of loop playback, update the current state to the new state, accumulate the training rounds and return to step 3 for iteration until the optimization rounds reach the preset total rounds, and the optimization is completed; Step 010: after the optimization is completed, enter the deployment stage, obtain the final resource configuration from the initial configuration through the greedy strategy based on the optimized strategy; and execute the scheduling decision module solving again based on the final resource configuration to obtain the final scheduling result.

[0015] Furthermore, the calculation process of the evaluation indicators in the resource allocation decision-making module includes: The sample mean and sample standard deviation are calculated based on the scheduling result set. Based on the sample mean and sample standard deviation, a comprehensive evaluation index is constructed using the comprehensive evaluation index formula. The sample mean and sample standard deviation are calculated based on the scheduling result set as shown in the following formula:

[0016]

[0017] in, The sample mean. The standard deviation of the sample is 1. For the number of sampling deployments, For current resource allocation The next The scheduling results for each sample; The formula for the comprehensive evaluation index is as follows:

[0018] in, As a comprehensive evaluation indicator, As an uncertainty penalty factor, ; When outputting the final resource configuration and scheduling results, select a set of candidate configurations with the top Q values ​​after training, re-solve the scheduling for each configuration, and select the configuration that meets the cost budget and has the shortest maximum completion time as the final resource configuration.

[0019] Secondly, the present invention provides an FMS scheduling and resource allocation joint optimization system, characterized in that it includes: The model building module is used for the actual production relationships based on the Flexible Manufacturing System (FMS), employing S with warehouse delay. 3 PR network constructs Petri net model of FMS, obtains S with place delay 3 PR network model, S with library delay 3 The PR net model represents the set of workpieces, process sequence, resource set, and time characteristics of each process in FMS. The resource set includes processing machines and transport robots. The scheduling decision module is used to combine the S with the warehouse delay. 3The PR network model, based on an initial or current resource configuration scheme, performs an improved Dyna-Q algorithm in a machine time-varying environment to obtain a maximum completion time set under multiple groups of time samples; the scheduling decision module includes scheduling strategy learning and scheduling strategy application; the scheduling strategy learning includes an outer loop and an inner loop, the outer loop randomly samples the processing time interval and the transportation time interval of each process to obtain time samples; the inner loop completes scheduling strategy learning based on the time samples through the improved Dyna-Q algorithm, and then delivers the scheduling decision results corresponding to each group of time samples to the resource configuration decision module through the scheduling strategy application; The resource configuration decision module is configured to analyze and evaluate the scheduling decision result set under multiple groups of time samples, calculate a reward function value, explore machine number configuration under a cost constraint, generate a new resource configuration scheme, and feed back to the scheduling decision module; The scheduling and resource configuration joint optimization module is configured to repeat the scheduling strategy learning and the scheduling strategy application of the scheduling decision module based on the new resource configuration scheme to obtain a new scheduling decision result set under multiple groups of time samples; the resource configuration decision module is repeatedly started based on the new scheduling decision result set under multiple groups of time samples, and joint optimization iteration is performed until a preset termination condition is met; the target of the joint optimization iteration is to minimize the system maximum completion time under the premise of meeting the resource and cost constraints; The production scheduling scheme generation module is configured to output the final resource configuration and scheduling result after the joint optimization iteration ends, sort the time axis according to the start time of each process of each workpiece, and generate an executable production scheduling scheme.

[0020] In a third aspect, the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the FMS scheduling and resource configuration joint optimization method when executing the computer program.

[0021] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the FMS scheduling and resource configuration joint optimization method.

[0022] Compared with the prior art, the present application has the following beneficial technical effects: The FMS scheduling and resource configuration joint optimization method provided by the application, in order to further improve the scheduling solving efficiency and the scheduling result quality, and solve the problem of solving the entire scheduling decision module which is relatively expensive under the evaluation of one resource configuration, the improved Dyna-Q algorithm is introduced in the scheduling and resource configuration, the Dyna-Q algorithm is a classical reinforcement learning framework, compared with the traditional algorithm such as meta-heuristic algorithm and the common reinforcement learning algorithm such as Q-learning, the real sample generated by the environment is used to update the value function, and the sample is stored in the internal model and 'virtually played back' several times on the model to further update, which is better at solving the problem under unknown environment parameters (such as machine processing time change, unknown number of machines), and can significantly improve the training efficiency and optimization strategy, a small amount of expensive sample can be exchanged for a large number of virtual updates, so as to significantly improve the learning efficiency and robustness on the actual problem of sample scarcity or simulation expense. Combined with the modeling method of PR (Systems of Simple Sequential Processes with Resources, resource simple sequential process system) net with the delay of the library under the Petri net, the improved Dyna-Q reinforcement learning algorithm can effectively solve the large-scale FMS scheduling problem under the change of machine processing time, and can effectively improve the solving quality of scheduling, and optimize the configuration scheme of resource configuration and improve the configuration training efficiency. The application can reasonably and efficiently solve the joint optimization problem of scheduling and resource configuration under complex environment, reduce the maximum completion time of system machine processing, improve the resource utilization and scheduling efficiency under the cost constraint, and is especially suitable for the optimization scene of multi-task complex flexible manufacturing system. 3 PR (Systems of Simple Sequential Processes with Resources, resource simple sequential process system) net modeling method, the improved Dyna-Q reinforcement learning algorithm can effectively solve the large-scale FMS scheduling problem under the change of machine processing time, and can effectively improve the solving quality of scheduling, and optimize the configuration scheme of resource configuration and improve the configuration training efficiency. The application can reasonably and efficiently solve the joint optimization problem of scheduling and resource configuration under complex environment, reduce the maximum completion time of system machine processing, improve the resource utilization and scheduling efficiency under the cost constraint, and is especially suitable for the optimization scene of multi-task complex flexible manufacturing system. BRIEF DESCRIPTION OF DRAWINGS

[0023] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. In addition, the shapes and scale sizes of the components in the drawings are only illustrative and are used to help understand the present application, and are not specific limitations on the shapes and scale sizes of the components.

[0024] In the drawings: Figure 1 The flow chart of the FMS scheduling and resource configuration joint optimization method of the application.

[0025] Figure 2 The structure diagram of the FMS scheduling and resource configuration joint optimization system of the application.

[0026] Figure 3 The electronic device schematic diagram of the FMS scheduling and resource configuration joint optimization method of the application.

[0027] Figure 4The flow chart of the FMS scheduling and resource configuration joint optimization method in the embodiment of the present application.

[0028] Figure 5 The structural diagram of the FMS scheduling and resource configuration joint optimization system in the embodiment of the present application.

[0029] Figure 6 The flow chart of the FMS scheduling and resource configuration joint optimization method in the embodiment of the present application.

[0030] Figure 7 The production requirement of the flexible manufacturing system in the embodiment of the present application.

[0031] Figure 8 The processing schematic diagram of the flexible manufacturing system in the embodiment of the present application.

[0032] Figure 9 The Petri net model diagram of the workshop environment of the flexible manufacturing system in the embodiment of the present application .

[0033] Figure 10 The multi-agent reinforcement learning flow chart based on the library delay Petri net in the embodiment of the present application.

[0034] Figure 11 The scheduling result Gantt chart of the flexible manufacturing system in the embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to enable the persons skilled in the art to better understand the present application scheme, the technical scheme in the embodiment of the present application will be described clearly and completely in combination with the drawings in the embodiment of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the persons skilled in the art without creative labor should belong to the protection scope of the present application.

[0036] Embodiment one Referring to Figure 1 , the FMS scheduling and resource configuration joint optimization method comprises the following steps: Based on the actual production relationship of the flexible manufacturing system FMS, the S 3 PR network with library delay is adopted to construct the Petri net model of the FMS, so as to obtain the S 3 PR network model with library delay, the S 3 PR network model with library delay represents the workpiece set, the process sequence, the resource set and the time characteristics of each process in the FMS, and the resource set comprises the machining machine and the transport trolley robot; The scheduling decision module is started, and the S 3The PR network model, based on an initial or current resource configuration scheme, executes an improved Dyna-Q algorithm in a machine time-varying environment to obtain a maximum completion time set under multiple groups of time samples; the scheduling decision module includes scheduling strategy learning and scheduling strategy application; the scheduling strategy learning includes an outer loop and an inner loop, the outer loop randomly samples the processing time interval and the transportation time interval of each process to obtain time samples, and the inner loop completes scheduling strategy learning based on the time samples through the improved Dyna-Q algorithm, and then delivers the scheduling decision results corresponding to each group of time samples to the resource configuration decision module through the scheduling strategy application; The resource configuration decision module is started to analyze and evaluate the scheduling decision result set under multiple groups of time samples, and to calculate the reward function value, to explore the number of machines under the cost constraint, to generate a new resource configuration scheme and to feed back the scheduling decision module; The scheduling strategy learning and the scheduling strategy application of the scheduling decision module are repeated based on the new resource configuration scheme to obtain a new scheduling decision result set under multiple groups of time samples; the resource configuration decision module is repeatedly started based on the new scheduling decision result set under multiple groups of time samples, and joint optimization iteration is performed until a preset termination condition is met; the target of the joint optimization iteration is to minimize the system maximum completion time under the premise of meeting the resource and cost constraints; After the joint optimization iteration ends, the final resource configuration and scheduling result are output, the time axis is sorted according to the start time of each process of each workpiece, and an executable production scheduling scheme is generated.

[0037] The modeling precision of this embodiment is outstanding, and the S 3 The PR network comprehensively characterizes workpieces, processes, processing / transportation resources and time characteristics, fits the actual production relationship of the FMS, and provides reliable model support for optimization. It has strong dynamic adaptability, the improved Dyna-Q algorithm is trained in a machine time-varying environment, can cover multiple groups of time samples, and ensures the robustness of the scheduling scheme in response to production fluctuations. The collaborative optimization is efficient, the double-process interactive iteration is performed, the resource configuration explores the number of machines under the cost constraint, the scheduling is simultaneously optimized, the process arrangement is realized, the deep collaboration of resource configuration and scheduling is realized, the constraint conditions are met, and the system maximum completion time is minimized. The output time axis sorting production scheduling scheme can be directly executed, has strong landing performance, can significantly improve the production efficiency of the FMS, reduce resource waste and production cost, and provides strong protection for the efficient operation of the flexible manufacturing system.

[0038] Embodiment two Referring to Figure 2 , the FMS scheduling and resource configuration joint optimization system comprises: The model establishment module is used for establishing a PR network based on the actual production relationship of the FMS, and the S 3The PR net constructs a Petri net model of the FMS, and obtains the S 3 The PR net model, the S 3 The PR net model represents the workpiece set, the process sequence, the resource set and the time characteristics of each process in the FMS, and the resource set includes machining machines and transport trolley robots. The scheduling decision module is used for combining the S 3 The PR net model, based on the initial or current resource configuration scheme, executes the improved Dyna-Q algorithm under the machine time change environment to obtain a set of maximum completion times under multiple groups of time samples; the scheduling decision module includes scheduling strategy learning and scheduling strategy application; the scheduling strategy learning includes an outer loop and an inner loop, the outer loop randomly samples the machining time interval and the transport time interval of each process to obtain time samples; the inner loop completes the scheduling strategy learning through the improved Dyna-Q algorithm based on the time samples, and then delivers the scheduling decision results corresponding to each group of time samples to the resource configuration decision module through the scheduling strategy application; The resource configuration decision module is used for analyzing and evaluating the scheduling decision result set under multiple groups of time samples, calculating the reward function value, exploring the machine number configuration under the cost constraint, generating a new resource configuration scheme and feeding back to the scheduling decision module; The scheduling and resource configuration joint optimization module is used for repeating the scheduling strategy learning and the scheduling strategy application of the scheduling decision module based on the new resource configuration scheme to obtain a new scheduling decision result set under multiple groups of time samples; the resource configuration decision module is repeatedly started based on the new scheduling decision result set under multiple groups of time samples, and joint optimization iteration is performed until a preset termination condition is met; the target of the joint optimization iteration is to minimize the system maximum completion time under the premise of meeting the resource and cost constraints; The production scheduling scheme generation module is used for outputting the final resource configuration and scheduling result after the joint optimization iteration ends, sorting the time axis according to the start time of each workpiece and each process, and generating an executable production scheduling scheme.

[0039] Embodiment three Referring to Figure 3 An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the FMS scheduling and resource configuration joint optimization method when executing the computer program.

[0040] Embodiment four A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the FMS scheduling and resource configuration joint optimization method.

[0041] Embodiment five Referring toFigure 4 The joint optimization method for FMS scheduling and resource allocation includes the following steps: Step 1: Based on the production relations of the actual flexible manufacturing system, model the flexible manufacturing system using a Petri net with warehouse-place delay. Establish the S-type of the flexible manufacturing system with warehouse-place delay under the Petri net according to actual production requirements. 3 PR network model; a practical flexible manufacturing system mainly includes various types of workpieces to be processed, multiple processing steps, and several parallel processing devices and shared resources. Based on actual production requirements, the workpieces to be processed are grouped together... To represent, where The first part represents the flexible manufacturing system. There are a total of [number] types of production workpieces. Type of production workpiece.

[0042] Each type of workpiece Consists of a set of ordered processes composition:

[0043] in, Indicates the workpiece being processed The One process, This represents the number of operations required to process the workpiece. The operations include machining by the processing machine and transportation by the transport robot.

[0044] Resource distribution in flexible manufacturing systems mainly includes two types: processing machines. and transport vehicle robots Its processing machines are used for It is said that the transport robot is used Indicates. Processing machinery. The transport robot is responsible for completing each processing step of each type of workpiece. Responsible for transferring the workpiece to the next processing machine. The number of each type of processing machine is... The number of each type of transport robot is The resource set of the entire flexible manufacturing system is represented as follows: .

[0045] Processing machines and transport robots both require a certain amount of processing and transport time during the processing and transportation processes. Each process... Available in candidate resource set Execute on any machine in the network.

[0046] Its processing machinery The processing time is a range of values:

[0047] Meanwhile, the processing steps require the use of transport robots before and after execution. The corresponding transportation time after loading or transshipment is completed is as follows:

[0048] Each processed workpiece The procedures must be completed in the given order:

[0049] Process The execution must meet the following conditions: 1. Previous process Completed; 2. Required processing machines Available; 3. The assigned transport robot idle.

[0050] The completion time for each machined workpiece is The maximum completion time (Makespan) of the system is:

[0051] The total resource cost of the system is subject to the budget. constraint:

[0052] The ultimate optimization goal is to minimize the maximum completion time while satisfying resource and cost constraints.

[0053] To better describe the concurrent processing, resource sharing, and mutual exclusion relationships in flexible manufacturing systems, this invention employs a Cubic-Position Delay Petri Net (SPON). 3 The PR model models this process. Through the formal definitions of places, transitions, and identifiers, this model can accurately depict the workpiece and resource allocation process in a flexible manufacturing system.

[0054] S 3 PR model is defined as Among them, the collection of warehouses , For the system to correspond to S 3 The number of places in the PR model represents the system state or resources, including active places, resource places, starting places, and ending places:

[0055] Among them, activity warehouse This indicates the intermediate state of a workpiece during a specific processing stage; the resource library contains... The token number in the table represents the available number of processing machines or transport robots, starting from the warehouse. This indicates that the current workpiece has not yet been put into processing, and the inventory period ends. It represents the state of the workpiece after processing.

[0056] Set of Changes , For the system to correspond to S 3 The number of transitions in a PR model represents the occurrence of system events or actions.

[0057] For flow relationship, The weight functions correspond to the order of each process in the system and the number of resources required, respectively.

[0058] Processing steps and the release and use of resources are characterized by the enabling emission of transitions, and the state of the system is identified by flags. It indicates that when changes occur Enabled (i.e., when all input libraries of the transition contain at least one token):

[0059] This allows for transition-enabled transmission and updates to the system state:

[0060] in Representing changes in Petri nets The preceding correlation matrix, Representing changes in Petri nets The post-transition correlation matrix. When a transition in the system is enabled, the weight values ​​on the corresponding directed arcs need to be consumed from all its preceding places, and the weight values ​​on the corresponding directed arcs need to be added to its subsequent places. This is used to characterize the completion and commencement of processing of the machine or transport robot in the actual flexible manufacturing system.

[0061] For each workpiece Each process sequence subnet needs to be constructed based on its processing sequence. After construction, a resource subnet needs to be constructed based on resource sharing within the processing sequence. Finally, these subnets are merged to complete the modeling. For the workpiece... Its actual processing sequence in the system is as follows:

[0062] in Characterizes the type of workpiece. Characterizing various processing paths for this type of workpiece. and This indicates the start and end of the machining process for this type of workpiece. express Workpiece of type 1 The first step in the process Processing path.

[0063] In the context of Petri nets, the starting library is used. and end of storage To characterize and The activity library represents the workpiece flow relationship along the processing path, and the resource library represents the resource release and use of the machines and transport robots required along the processing path, represented as: , After all processing paths for all types of workpieces are constructed, all process sequence subnets and resource subnets are merged to obtain the final network model.

[0064] To characterize the time cost required for each step in the processing path, in each activity warehouse Additional time range for the meeting , Indicates the first Type of workpiece in the first The first step in the process The processing time required for each processing path is not appended to the starting warehouse, ending warehouse, and resource warehouse.

[0065] Step 2: Activate the scheduling decision module. In this embodiment, a scheduling agent is used to train and deploy under varying machine time conditions based on the initial or current resource configuration scheme. This obtains the maximum completion time set across multiple time samples under the current configuration scheme, and the results are fed to the resource configuration decision module's resource configuration agent. The scheduling agent is used to make sequential decisions regarding the processing and transportation processes in the flexible manufacturing system under given resource configuration conditions. This agent takes the system's operating state based on Petri nets as input, selects scheduling actions from currently executable processing or transportation operations, and continuously optimizes the scheduling strategy through reinforcement learning to obtain a smaller maximum completion time. The resource configuration agent is used to determine the configuration quantity of resources such as processing machines and transportation robots in the flexible manufacturing system under cost constraints. This agent adjusts the resource configuration scheme based on the scheduling results feedback, and gradually learns resource configuration strategies that can improve the overall scheduling performance of the system through reinforcement learning.

[0066] See Figure 10A Dyna-Q tabular multi-agent reinforcement learning algorithm is used for joint optimization of scheduling and resource allocation. For the constructed time-delay Petri net model, an initial resource allocation scheme is given, and the number of processing machines and transport robots in the resource allocation scheme are represented on the resource pool in the time-delay Petri net, that is, on the corresponding resource pool. The corresponding number of tokens are added to it.

[0067] The training and deployment of the scheduling agent begins based on the current configuration scheme. The final deployment result of the scheduling agent is a set of independent and identically distributed scheduling results (maximum completion time). These results are then input back to the resource allocation agent for sample evaluation, training, and deployment, thereby obtaining a new resource allocation scheme. This process continues iteratively until the joint optimization of scheduling and resource allocation in the flexible manufacturing system is completed, providing a reasonable and excellent resource allocation scheme and the scheduling results under that scheme.

[0068] Based on the final resource allocation and scheduling results, the operations are sorted along a timeline according to the start time of each workpiece and each process, generating an executable production scheduling plan.

[0069] The training and deployment process in a scheduling agent includes the following steps: The nominal state of the constructed system is The system sampling status is ,in For Petri net state identifiers, List of nominal remaining time, Used for scheduling state Q-value updates during training. It does not change with the actual remaining processing time, but rather takes the median of the processing time interval. This represents the actual remaining processing time after each action selection following the current machine processing time sampling. It sets the initial state. and target state .

[0070] After obtaining the configuration scheme output by the resource configuration agent, the two-layer loop training of the scheduling agent begins: scenario sampling. The outer loop (outer layer) and the inner loop (sample training layer) are used for sampling. The outer loop primarily samples the processing time intervals of each machine, and the total number of outer loop iterations is set. Each cycle uses a specific set of processing time sampling samples. Processing time sampling This includes the processing time of the processing machine and the transportation time of the transport robot. These time samples are then input into the inner loop for training. Each loop operates within the time delay interval of each resource repository. Random sampling is performed to obtain the current Time Sample The sample is then input into the inner loop for training. The inner loop primarily samples the current processing time along with other system information such as the workpiece being processed and the processing steps. It uses an improved Dyna-Q algorithm for training and deployment. After the inner loop finishes, the current scheduling strategy and results are retained, and the next outer layer training is performed. After both inner and outer layer training are completed, the scheduling agent enters the deployment phase after the training phase, where the trained Q_s table is... The second sampling deployment will The results obtained from each sampling deployment are recorded. After the sampling deployment is completed, the final scheduling policy (Q_s table) and... The set of scheduling results corresponding to each time sample under the sub-sampling deployment (the maximum completion time under each time sample) is used as the output of the scheduling agent.

[0071] The deployment results of the scheduling agent are output to the resource allocation agent, mainly to calculate the reward function of the resource allocation agent, thereby optimizing the resource allocation scheme.

[0072] Step 3: After obtaining the output results of the scheduling agent, the resource allocation agent analyzes and evaluates the data, calculates the reward function value, explores the number of machines under the given cost constraint, and provides the new resource allocation scheme to the scheduling agent for training and deployment.

[0073] Step 4: The scheduling agent and the resource allocation agent jointly train through interactive iteration. The resource allocation agent provides the resource configuration, and the scheduling agent trains under the resource configuration and provides the scheduling results for the resource allocation agent to analyze. The two interact and update in a loop until the preset termination condition is reached.

[0074] Step 5: After the multi-agent interactive iterative joint training is completed, the final resource configuration and scheduling results are exported. The operations are sorted on the timeline according to the start time of each workpiece and each process to generate an executable production scheduling plan.

[0075] The improved Dyna-Q algorithm used in the inner training loop includes the following steps: Step 001: Set the initial learning rate for scheduling. Discount rate Scheduling replay threshold Scheduling planning steps Total number of training rounds The learning rate decay coefficient of the outermost loop Inner loop learning rate decay coefficient Attenuation ratio Exploration rate return temperature difference Forgetting rate coefficient Number of times the sampling deployment is scheduled Scheduling decay factor The scheduler deadlock penalty is penalty_s, the scheduler target reward is reward_s, and the scheduler policy table Q_s, the scheduler replay list pqueue_s, the scheduler transition list model_s, and the scheduler predecessor list pred_s are initialized. Step 002: Initialize the identifier , After sampling the current machine processing time, the actual remaining processing time after each action selection. ; Step 003: Based on the current identifier Find the set of enableable transitions under the current Petri net and adopt an improved global exploration rate. A dynamic greedy strategy selects a transition to enable as the current state. The following action Received a new identifier New nominal status and new sampling status The nominal remaining time in the new state The remaining sampling time in the new state is represented by the median of the machine processing interval. Machine processing time based on actual sampling Perform calculations; Improved global exploration rate in the Dyna-Q algorithm under a scheduling agent The greedy strategy is designed as follows: Set global exploration rate initial value Global exploration rate final value ; During the outer training loop of each scenario sampling, the inner exploration rate... Employing an exponential decay strategy, the inner layer... The exploration rate of each round , It is the attenuation factor; After completing the inner loop, a warm-up process is required before entering the next outer scenario sampling. This means that whenever entering a new outer scenario... A brief rebound is needed To explore new scenarios ,in This is the temperature difference. This is the exploration rate at the end of the previous inner loop.

[0076] Step 004: Calculate Instant Rewards When the new logo When no enableable transitions are available for action selection, the new state is considered to have entered a deadlock. A negative scheduling deadlock penalty penalty_s; when the new identifier Equal to target identifier At this time The reward is reward_s for a positive scheduling target; in general scenarios, pass The calculation is performed; the general scenario is that after performing an action (launch transition), the new state does not enter a deadlock and is not the target state. In this case, the reward function value needs to be calculated using the current list of actual remaining time.

[0077] Examples of instant rewards in general scenarios Calculation: Assuming the current state, the selected action The corresponding strain change All Pre-Activity Library , , And given that the current actual remaining time for each activity library is 2.4, 1.3, and 1.9 units of time, respectively, then the action in the current state must be completed. Afterwards, the remaining time of all preceding activities in the transition corresponding to the action will be reset to zero, and the largest remaining time will be selected as the immediate reward for the current action (if the new state is not a deadlock state). In this example, 2.4 units of time will be used as the current state to take the action. Instant rewards .

[0078] Step 005: Transfer the state transition sample The state transition is recorded in the scheduling state transition module mode_s, and the set of state predecessors is also recorded in the scheduling predecessor list pred_s. Fill in ; Step 006, according to Time difference error Calculated, such as Not deadlocked and Greater than the scheduling replay threshold Then experience Push it into the min-heap scheduling replay list pqueue_s, and then use an improved adaptive decay learning rate. pass Update the Q_s table once; Improved adaptive decay learning rate in the Dyna-Q algorithm under a scheduling agent The design is as follows: Because we want the model to learn quickly in new scenarios, the learning rate in the Dyna-Q algorithm is... An adaptive decay strategy is adopted when entering a new The effect is brief, followed by a gradual decline:

[0079] in r represents the number of iterations in the outer loop. Represents the number of inner loop iterations. This represents the initial learning rate or the learning rate at the end of the previous inner / outer loop. and This represents the decay coefficient of the learning rate in the outer and inner loops; Step 007, proceed The scheduling loop performs a virtual replay, popping the highest-priority entry from the min-heap scheduling replay list pqueue_s in each iteration. The priority of the design improvement is to be with Size and current outer are related. The scheduling state transition module model is used to find... Find by referring to the Q_s table The Q value corresponding to the optimal action. Substitute this into the Q-value update formula in step 006 to update the value. Then find the scheduling predecessor set. The set of states, calculating the state of all predecessor states. Greater than the scheduling replay threshold Entries Similarly, push the data into the min-heap scheduling replay list pqueue_s. Repeat this process until completion. Virtual replay of the next scheduling cycle; The priority design of the replay list in the Dyna-Q algorithm under the scheduling agent is as follows: To emphasize the training effect under the new time samples, when considering A decay weight is added along with the error, increasing the likelihood of new experience entries being replayed while still allowing older ones to be replayed. Each experience entry is recorded with its outermost number, which is used to calculate its decay weight. :

[0080] in, This is the forgetting rate coefficient. and This refers to the current outer layer number and the outer layer number corresponding to this experience entry.

[0081] The priority of each empirical item is calculated using attenuation weights. ,in As a decay ratio, the highest priority experience entry will be placed at the top of the min-heap scheduling replay list pqueue_s; Step 008: Update the nominal state and the sampled state, then return to step 003 to continue the loop until... Reaching the target state Alternatively, if a deadlock occurs, end the current training round and return to step 002 to begin a new training round. The total number of training rounds has been reached. The outer loop training ends at the specified time; Step 009: When all outer loops are scheduled After training, the trained Q_s table is processed. The secondary scheduling sampling deployment follows the same sampling process as the time sample sampling process in the outer sampling layer. From the initial state... Start by finding the transition with the largest Q value in the current state as the action, and record the reward for each action. And it accumulates until the target state is reached. (Rewards for reaching the target state) No need to add positive scheduling target reward It's important to note that the Q_s table is frozen and not updated during the sampling deployment process. The final total reward... The maximum completion time (makespan) obtained by scheduling the agent is the action sequence, which is the machine processing sequence. The results obtained from the next scheduling sampling deployment are recorded. After the sampling deployment is completed, the set of scheduling results (deployment results) and the final scheduling policy (Q_s table) are used as the output of the scheduling agent.

[0082] Here is a brief explanation of the sampling deployment: After completing the outer sampling and inner training loops, the training phase of the scheduling agent ends, and the deployment phase begins. The goal of the deployment phase is to obtain the scheduling results under the current resource configuration scheme. Since the machine processing time and the transport robot's transport time in the current system are given as time interval information, it is necessary to provide the scheduling results of multiple independent and identically distributed time sampling samples in order to comprehensively represent the scheduling information under the current resource configuration scheme, thereby better enabling subsequent resource allocation.

[0083] Specifically, the training and deployment process of the resource allocation agent includes the following steps: Step 101: Initialize the configuration strategy table Q_a, the replay list pqueue_a, the transition list model_a, and the predecessor list pred_a, and set the learning rate. Configure discount rate Configure playback threshold Configure the number of planned steps Configure the total number of training rounds Initial and final exploration rates , Cost penalty coefficient Cost budget ceiling Configure attenuation factor Configure deadlock penalties Penalties for overspending Uncertainty penalty factor Configure the number of deployments ; Step 102: Enter the loop iteration and set the current state. For initial resource configuration ,state Set as a resource configuration vector It represents the number of various processing machines and transport robots; Step 103: Obtain the current resource configuration by solving the scheduling decision module. The following is a set of scheduling results ,in This represents the number of scheduling results in the set, i.e., the number of results obtained by the scheduling agent after sampling and deployment; Step 104: For the current state List the legal action set The actions Characterized as the first The number of machines can be increased or decreased by one unit, based on the upper limit of the cost budget. Restricted action set .according to Greedy strategy selects action The first step in the training process exploration rate in each round Execute actions Acquire new state ; Step 105: Calculate the evaluation indicators It represents the current resource allocation scheme. Select action The size and stability of the subsequent scheduling results. Calculation of reward payout:

[0084] in This represents the total cost under the new resource allocation plan. If the new resource allocation plan is unreasonable and leads to budget overruns, a penalty will be imposed. Then revert the new state to the original state. And return to step 104; Calculate evaluation indicators Includes the following steps: Step 1051: Based on resource allocation The following is a set of scheduling results Calculate its sample index: Sample mean:

[0085] Sample standard deviation:

[0086] Step 1052: Construct a comprehensive evaluation index for the scheduling result set. :

[0087] in, Controlling the punishment of uncertainty; Step 106: Record the state transition relationships using the configuration transition list model_a and the configuration predecessor list pred_a; Step 107: Update Q value , like error Greater than the configured replay threshold Then Push the configuration replay list pqueue_a into the min-heap; Step 108, proceed In each loop replay, the highest priority entry in the min-heap configuration replay list pqueue_a is popped. And find the configuration state transition module model_a Find the table Q_a The Q value corresponding to the optimal action. Substitute this into the Q-value update formula in step 107 to update the value. Then find the front-drive configuration based on the set of configurations. The set of states is used to calculate the time limit (td) of all predecessor states, which is greater than the replay threshold. Entries Push the configuration replay list pqueue_a into the min-heap. Repeat this process until completion. This configuration allows for virtual loop replay; Step 109: Update Status Add one training round, return to step 103 and continue iterating until the set total number of training rounds is reached. Training is over; Step 110: Enter the deployment phase of the resource allocation agent, based on the trained strategy. Starting from the initial configuration, a final configuration is obtained by following a greedy strategy; The training and deployment of the resource allocation agent have now been completed. Based on the final resource allocation c*, the scheduling decision module is solved again. During this solution process, the warehouse delay is fixed from the interval to the median of the interval to obtain the scheduling result corresponding to the final configuration.

[0088] At the same time, select the first Add the configuration with the highest Q value to the candidate configuration set (if there is overlap, select only one); Step 1011: Perform scheduling and solution on each candidate configuration, and select the configuration with the smallest maximum completion time that meets the cost budget requirements as the final configuration.

[0089] The following description, in conjunction with the accompanying drawings, further illustrates this embodiment: See Figure 6 The joint optimization method for FMS scheduling and resource allocation includes the following steps: Based on actual production requirements, the S-value of the Petri network-based storage delay is used. 3 PR network models the flexible manufacturing system; the warehouse delay S of the flexible manufacturing system 3 The PR net model includes a set of types and numbers of components to be processed, a set of processing machine resources, a set of processing paths, and a set of machine processing times. The dynamic behavior of the system is defined by initial identifiers and state transition formulas. Taking into account variations in machine processing time, an improved Dyna-Q algorithm is used for training to obtain the maximum completion time under the current resource configuration. The results obtained by the scheduling decision module are used as the reward for each action. An improved Dyna-Q algorithm is used for training to continuously optimize the resource allocation scheme and obtain the optimized resource allocation scheme. The final scheduling result is generated based on the optimized resource allocation scheme and machine processing time.

[0090] Specifically as follows: Step 1111: Construct a time-delay Petri net model based on a flexible manufacturing system example; Depend on Figure 8 It can be seen that there are two types of workpieces in the current system. , Processing is required, and the processing sequence for each workpiece is given: For workpiece one In general, it takes five processes (including transportation and processing) to complete the processing of this workpiece, and the order of these processes cannot be changed. Workpiece 1 It needs to pass through the No. 1 transport robot first. After the corresponding transportation process, on the No. 1 processing machine Or the No. 2 processing machine The corresponding processing steps are completed on the first stage, and then the first transport robot passes through it. After completing the corresponding transportation process, at processing machine number three... After completing the corresponding processing steps, it passes through the No. 2 transport robot. After completing the corresponding transportation process, the workpiece is completed. All processing work.

[0091] Similarly, for workpiece two For example, it needs to first pass through the No. 2 transport robot. After the corresponding transportation process, it is processed on machine number four. The corresponding processing steps are completed on the first stage, and then the first transport robot passes through it. After completing the corresponding transportation process, at processing machine number two... After completing the corresponding processing steps, it passes through the No. 1 transport robot. After completing the corresponding transportation process, complete workpiece two. All processing work.

[0092] Figure 9 for , The processing diagrams for the two types of workpieces clearly show that there are multiple processing paths, multiple processing elements, non-unique numbers of transport trolleys and processing machines, resource sharing, and resource occupation conflicts during the processing of the two types of workpieces. Figure 9 middle The outer circle represents the storage area, and the inner circle represents the storage area. Indicates Token, It indicates change.

[0093] Based on current examples of flexible manufacturing systems, transport robot , Use resource library , This indicates that the processing machine , , , Use resource library , , , The number of its transport robots and processing machines is represented by tokens in the resource repository. (Using the starting repository) , Characterization , Two types of workpiece start processing buffer zones, with end storage. , Characterization , Two types of workpiece end-of-process buffer zones. The workpiece's state during the completion process is represented by an active library. To characterize, such as for workpiece one When it performs its first process (the No. 1 transport robot) Transportation) corresponding activity warehouse If there is no Token in the workpiece, then the workpiece is considered to be one. The first process has not yet been carried out and the workpiece is still within the initial processing buffer. If the active warehouse contains a token and the token currently has remaining processing (transportation) time, then the workpiece is considered to be in the initial processing buffer. In the process of executing the procedure (Robot No. 1 transport vehicle) Workpiece is being transported If the remaining processing (transportation) time of the workpiece reaches zero, it means that the workpiece is ready for transport. The first process has been completed and is awaiting the second process (processing machine number one). Or the No. 2 processing machine The execution of processing steps.

[0094] In the starting, ending, and resource warehouses, the number of tokens in each warehouse represents the number of processing elements contained in the transport trolleys or processing machines in the current process. Its size is constrained by the number of tokens in the corresponding resource warehouse (representing the number of transport trolleys or processing machines, given by resource allocation). Each token in a warehouse has a remaining processing (transport) time, which represents the processing (transport) time required for the processing (transport) element to complete the current processing (transport) process.

[0095] Because the machining (cart transport) time is not constant in actual production and often fluctuates within a certain range (randomness or affected by workpiece / tool / environment), a time delay interval is added to each resource library to characterize the time taken for machining (cart transport) in the actual machining (transport) process. For example... Figure 7 In this example, the time delay interval satisfies the mapping:

[0096]

[0097]

[0098]

[0099] The cost information for each resource is mapped as follows:

[0100]

[0101] Through changes The enabling transitions are used to realize the transfer of tokens between warehouses and places, which represents the selection of a specific processing machine (cart) to perform the processing (transportation) operation during the actual processing (transportation) process. In this example, the directed arc weights between warehouses and transitions are all 1, representing that a processing machine (cart) on each operation of various processed workpieces can only process (transport) one component at a time. The enabling transitions corresponding to the processing machines (carts) are shown in the table below.

[0102] The machining path of a workpiece is characterized by locations and transitions: Workpiece 1:

[0103] or

[0104] Workpiece 2:

[0105] After analyzing and modeling the production requirements of the flexible manufacturing system instance, multi-agent reinforcement learning training and deployment are carried out based on the modeled time-delay Petri net model under the current instance, and finally a set of resource allocation schemes and scheduling results under the schemes are obtained.

[0106] Multi-agent reinforcement learning mainly consists of two parts: a scheduling agent (scheduling solver) and a resource allocation agent. The scheduling agent and the resource allocation agent jointly train through interactive iteration. The resource allocation agent provides the resource allocation, and the scheduling agent trains under the resource allocation and provides the scheduling results for the resource allocation agent to analyze. The two interact and update cyclically until a preset termination condition is reached, and the final resource allocation and scheduling results are derived.

[0107] Step 1112: Use the Dyna-Q tabular multi-agent reinforcement learning algorithm to jointly optimize scheduling and resource allocation; For a pre-constructed Petri net model of place delay, given a set of initial resource configuration parameters... Initial identifier The target is identified as .

[0108] First, we will proceed with the training and deployment of the resource allocation agent: 1) Initialize configuration strategy Table _a, Configuration Replay List Configure transfer list and front-wheel drive configuration list Configure learning rate Set the discount rate to 0.05. The value is 0.98, and the total number of training rounds is configured. The initial and final exploration rates are 2000. , The cost penalty coefficients are 0.8 and 0.05. The upper limit of the cost budget is 100. Configure the attenuation factor to 25000. Set the value to 0.995 and configure deadlock penalty. The penalty for exceeding the budget is 1000. The uncertainty penalty factor is 5000. Set the number of deployments to 10. It is 7; 2) Use the current state as the initial state of the resource allocation agent. ,according to Greedy strategy selects one action And execute, assuming that processing machine number one has been selected. Performing the increment action will obtain the new resource configuration. And corresponding to the new state Calculate the new state The corresponding cost: Therefore, it meets the resource cost constraints and can be used for subsequent scheduling solutions (such as a certain resource allocation). The corresponding cost is At this time, resource allocation should be maintained. Without changing the reward function and without performing subsequent scheduling, let it remain unchanged. (The status is not updated). 3) Provide a new initial identifier for the Petri net based on the new resource configuration. and new target identifiers Based on this, the scheduling agent under this resource configuration is trained and deployed: 3.1) Set the total number of outer loop iterations The total number of inner scheduling loops is 10. The initial learning rate is set to 10000. Set to 0.08, scheduling discount rate The threshold for scheduling replay is 0.98. The number of scheduling planning steps is 0.01. The learning rate decay coefficient of the outer loop is 20. The learning rate decay coefficient of the inner loop is 0.01. The exploration rate recovery temperature difference is 0.001. The forgetting rate coefficient is 0.2. The number of times the sampling deployment is 0.3 is scheduled. It is 7; 3.2) Enter the outer loop and perform time sampling (with a precision of 0.1) to obtain the current time sample of the outer loop:

[0109]

[0110]

[0111]

[0112] 3.3) Enter the inner loop and determine the current state flag. by Find the set of enableable transitions (action space): To improve Dynamic Greedy Strategy Selection As the current state The following action Obtain a new status identifier ,in , ; 3.4) Calculate instant rewards Due to the change in the chosen action Pre-position warehouse ( If there is no delay, then ; 3.5) State transition samples Record the changes in the scheduling transfer module model_s and the list of scheduling predecessors pred_s; 3.6) Calculation error: Therefore, experience Instead of pushing the data into the min-heap, replay the data and perform a Q-table update. Since the current experience cannot demonstrate the complete scheduling and training process, it is assumed here that an experience was found in a certain loop. Play back the video; in this experience After priority calculation, this experience has the highest priority and is therefore located at the top of the min-heap replay list. During the replay process, the Q-value is updated again, and the experiences of all its predecessor states are found and their values ​​are calculated. After adding to the replay list, continue replaying until the maximum number of replays is reached; 3.7) Update Status The identifier , , ; 3.8) Repeat steps 3.3) to 3.7) until the state reaches the target state or a deadlock state, then the inner loop ends. After completing the specified number of inner loops, the next outer sampling loop begins. 3.2) Repeat this process until the outer loop count is reached, then the scheduling agent training phase ends. 3.9) After the training phase is completed, the current resource allocation The following sampling deployment results :

[0113]

[0114] 4) Calculate the new resource configuration based on the sampling deployment results. Evaluation indicators And calculate the resource allocation scheme. Select action After obtaining the state Rewards and benefits :

[0115]

[0116]

[0117]

[0118] 5) Apply state transition experience Record the configuration transfer module model_a and the configuration predecessor set list pred_a; 6) Update the Q_a table once based on current experience, which is basically similar to 3.6) in the scheduling agent, and will not be repeated here; 7) Perform experience replay and update the Q_a table again, which is basically similar to 3.7) in the scheduling agent, so it will not be described in detail here; 8) Update status Add one training round, return to step 3) and continue the loop iteration until the total number of training rounds is reached as configured. The training phase of the resource allocation agent has ended; 9) Based on the trained strategy Starting from the initial configuration, a final configuration scheme is obtained by following a greedy strategy: The cost of 22920 meets the cost ceiling. Using this configuration scheme, the scheduling agent is used to solve the problem, and the scheduling result is obtained: Transition sequence:

[0119]

[0120] Maximum completion time: 28 Step 1113: Export the final resource configuration scheme and scheduling scheme, and give the machine processing time based on the current scheduling scheme; The trigger times for each transition, obtained from the transition sequence based on the scheduling results, are shown in the table below:

[0121] The corresponding Gantt chart of scheduling results in a real flexible manufacturing system is as follows: Figure 11 As shown, 1-1, 1-2, 1-3 and 2-1, 2-2, 2-3 respectively represent as The first, second, and third workpieces and The first, second, and third workpieces.

[0122] See Figure 5 The FMS scheduling and resource allocation joint optimization system includes: The model building module uses the S-type delay of the Petri net with a storage area based on actual production requirements. 3 PR network models the flexible manufacturing system; the warehouse delay S of the flexible manufacturing system 3 The PR net model includes a set of types and numbers of components to be processed, a set of processing machine resources, a set of processing paths, and a set of machine processing times. The dynamic behavior of the system is defined by initial identifiers and state transition formulas. The scheduling agent module, taking into account the changes in machine processing time, is trained using an improved Dyna-Q algorithm to obtain the maximum completion time under the current resource configuration. The resource allocation agent module uses the results solved by the scheduling decision module as the reward for each action, and is trained using an improved Dyna-Q algorithm to continuously optimize the resource allocation scheme and obtain the optimized resource allocation scheme. The final scheduling result generation module is used to generate the final scheduling result based on the optimized resource configuration scheme and machine processing time.

[0123] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0124] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for joint optimization of FMS scheduling and resource configuration, characterized in that, The method comprises the following steps: Based on the actual production relationship of the Flexible Manufacturing System (FMS), an S-type system with warehouse delay is adopted. 3 The PR network constructs the Petri net model of the FMS to obtain the S with place delay. 3 PR network model, the S with library delay 3 The PR net model represents the set of workpieces, process sequence, resource set, and time characteristics of each process in FMS. The resource set includes processing machines and transport robots. The scheduling decision module is started, and S 3 The PR network model, based on the initial or current resource configuration scheme, executes the improved Dyna-Q algorithm under the machine time-varying environment to obtain a maximum completion time set under multiple groups of time samples; the scheduling decision module includes scheduling strategy learning and scheduling strategy application; the scheduling strategy learning includes an outer loop and an inner loop; the outer loop randomly samples the processing time interval and the transportation time interval of each process to obtain time samples; the inner loop completes the scheduling strategy learning through the improved Dyna-Q algorithm based on the time samples, and then delivers the scheduling decision results corresponding to each group of time samples to the resource configuration decision module through the scheduling strategy application. starting a resource configuration decision module, analyzing and evaluating a scheduling decision result set under multiple groups of time samples, and calculating a reward function value, exploring machine number configuration under a cost constraint, generating a new resource configuration scheme, and feeding back the scheduling decision module; repeating scheduling strategy learning and scheduling strategy application of the scheduling decision module based on the new resource configuration scheme to obtain a new scheduling decision result set under multiple groups of time samples; starting the resource configuration decision module again based on the new scheduling decision result set under multiple groups of time samples, and iteratively optimizing jointly until a preset termination condition is met; the target of the iterative joint optimization is to minimize the maximum completion time of the system under the premise of meeting the resource and cost constraints; after the iterative joint optimization ends, outputting a final resource configuration and scheduling result, sorting a time axis according to the start time of each workpiece and each process, and generating an executable production scheduling scheme.

2. The FMS scheduling and resource configuration joint optimization method according to claim 1, characterized in that, The scheduling strategy learning of the scheduling decision module specifically comprises the following steps: constructing a system nominal state and a system sampling state; the system nominal state and the system sampling state both comprise a Petri net state identifier and corresponding processing remaining time; the system nominal state corresponds to nominal remaining time, and the system sampling state corresponds to actual processing remaining time after sampling of current machine processing time and selection of each action; meanwhile, an initial state and a target state of the system are set; obtaining an initial or current resource configuration scheme output by the resource configuration decision module, and performing outer loop training and inner loop training of the scheduling decision module; the outer loop training comprises sampling of a processing time interval of each machine, setting a total number of outer loop cycles, using a group of processing time sampling samples in each outer loop cycle, and inputting the processing time sampling samples to the inner loop for sample training; the inner loop training comprises scheduling strategy learning and scheduling strategy application of samples by using an improved Dyna-Q algorithm according to the current processing time sampling samples and system information, obtaining a scheduling strategy and a result of the scheduling decision module, and retaining the trained scheduling strategy and result of the scheduling decision module after the inner loop ends and performing next outer loop training; The scheduling strategy application of the scheduling decision module specifically comprises the following steps: after the outer loop training and the inner loop training of the scheduling decision module end, sampling and deploying a Q_s table of the trained scheduling decision module multiple times, recording results obtained by multiple sampling and deployments, and obtaining a final Q_s table and a maximum completion time set under each time sample after the sampling and deployment end.

3. The FMS scheduling and resource configuration joint optimization method according to claim 2, characterized in that, The improved Dyna-Q algorithm comprises the following steps: Step 1: initializing scheduling parameters, a scheduling strategy table, a scheduling replay list, a scheduling transition list, and a scheduling predecessor list; Step 2: initializing a current identifier, nominal remaining time, and actual processing remaining time after action selection; Step 3: based on the current identification obtained in step 2, determining an enabled transition set under the Petri net, and adopting an improved global exploration rate The greedy strategy selects the transition enablement as the action in the current state, obtains a new identification and corresponding nominal state, sample state and new state, the new state is the set of the nominal state and the sample state, and generates a state transition sample; Step 4: judging a state type according to the new identifier obtained in step 3, and calculating an immediate reward according to the state type; the state type comprises a deadlock state, a target identifier state, and a general state, and different state types correspond to different immediate reward calculation modes; Step 5: record the state transition sample generated in step 3 to the scheduling transition list, and record the current state as the predecessor state of the new state obtained in step 3 and store it in the scheduling predecessor list; Step 6: calculate the temporal difference error by multiplexing the immediate reward in step 4 and the new state in step 3, and if the new state is not a deadlock state and the temporal difference error meets the preset condition, store the corresponding experience in the scheduling replay list, and update the scheduling policy table using the improved adaptive decay learning rate; Step 7: perform a preset number of virtual replays, extract the highest priority experience in the scheduling replay list each time, determine the new state corresponding to the highest priority experience based on the scheduling transition list constructed in step 5, and update the policy value corresponding to the highest priority experience based on the scheduling policy table; at the same time, process the temporal difference error of the predecessor state based on the scheduling predecessor list in step 5, and store the experience meeting the condition in the scheduling replay list; Step 8: after updating the nominal state and the sampling state obtained in step 3, return to step 3 for continuous circulation until the new state is the target state or a deadlock state to end this round of optimization; repeat the above optimization process until the preset total optimization round is reached to complete the outer loop optimization; Step 9: after completing all outer loop training in step 8, perform a preset number of sampling deployments on the trained scheduling policy table; start from the initial state, select the action with the optimal policy value to execute, record the reward of each action and add it to the target state, the scheduling policy table remains frozen during the sampling deployment process, and finally output the sampling deployment result set and the scheduling policy table.

4. The FMS scheduling and resource configuration joint optimization method according to claim 3, characterized in that, the improved global exploration rate a greedy strategy, in particular: Set global exploration rate Initial value is Global exploration rate final value ; In each outer training loop of episodic sampling, the inner global exploration rate With an exponential decay strategy, the inner exploration rate of the is the decay factor;​​ After the inner loop is completed, the temperature is raised when entering the next outer loop scenario sampling, and the temperature is raised whenever a new cycle is entered to explore new scenarios where is the temperature difference, is the exploration rate at the end of the previous inner loop The adaptive decay learning rate is as follows: wherein, represents the adaptive decay learning rate, r represents the outer loop number of iterations, represents the inner loop number of iterations, represents the initial learning rate or the learning rate at the end of the previous inner / outer loop, and represents the decay factor of the learning rate in the outer and inner loops.

5. The FMS scheduling and resource configuration joint optimization method according to claim 3, characterized in that, The priority of the replay list of the improved Dyna-Q algorithm considers the timing difference error At the same time, the decay weight is added, the priority of each experience entry is calculated by using the decay weight, the experience entry with the maximum priority is placed at the top of the min-heap replay list, and the decay weight is as follows: wherein, is a decay weight, is a forgetting speed coefficient, and is a current outer layer number and an outer layer number corresponding to the experience entry.

6. The FMS scheduling and resource configuration joint optimization method according to claim 1, characterized in that, The resource configuration learning process of the resource configuration decision module includes the following steps: Step 01: initialize the configuration policy table, the configuration replay list, the configuration state transition list and the configuration predecessor list of the resource configuration, and set the training parameters to obtain the initial resource configuration; Step 02: after initialization, enter the loop iteration, take the initial resource configuration as the initial state, and the initial state represents the number configuration of various processing equipment and transportation equipment; Step 03: based on the resource configuration corresponding to the current initial state, solve through the scheduling decision module to obtain the scheduling result set under the current resource configuration; Step 04: after obtaining the scheduling result set, determine the legal action set for the current state, the legal action set is limited by the cost budget; select and execute the action based on the greedy strategy to obtain a new state; Step 05: after obtaining the new state, calculate the evaluation index and reward return corresponding to the new state; if the resource configuration corresponding to the new state exceeds the cost budget, apply a penalty and return the state to the original state, and return to step 4 to select an action again; Step 06: after completing the reward return calculation and state rollback processing, record the state transition relationship through the configuration state transition list and the configuration predecessor list; Step 07: after recording the state transition relationship, update the Q value; if the temporal difference error exceeds the preset threshold, push the state action information into the configuration replay list; Step 08: After completing the Q value update and pushing the configuration playback list item, a preset number of loop playback is performed, each time the highest priority item is popped out from the configuration playback list, the subsequent state is determined based on the configuration state transition list, the optimal action corresponding to the Q value in the subsequent state is found combined with the configuration strategy table, and the Q value of the current item is updated; at the same time, the predecessor state set of the current state is determined according to the configuration predecessor list, the time sequence difference error of each predecessor state is calculated, and the predecessor state information meeting the threshold condition is pushed into the configuration playback list until the preset number of loop playback is completed; Step 09: After completing the preset number of loop playback, the current state is updated to the new state, the training round is accumulated and returned to step 3 for iteration until the optimization round reaches the preset total round, and the optimization is completed; Step 010: After the optimization is completed, enter the deployment phase, obtain the final resource configuration from the initial configuration based on the optimized strategy through the greedy strategy; based on the final resource configuration, the scheduling decision module is executed again to obtain the final scheduling result.

7. The FMS scheduling and resource configuration joint optimization method according to claim 6, characterized in that, The calculation process of the evaluation index in the resource configuration decision module includes: Based on the scheduling result set, the sample mean and sample standard deviation are calculated, and the comprehensive evaluation index is constructed through the comprehensive evaluation index formula according to the sample mean and sample standard deviation; The calculation of the sample mean and sample standard deviation based on the scheduling result set is as follows: wherein, is the sample mean, is the sample standard deviation, is the number of sample deployments, is the current resource configuration the scheduling result of the th sample under the current resource configuration; The comprehensive evaluation index formula is as follows: wherein, is a comprehensive evaluation index, is an uncertainty penalty factor, ; When outputting the final resource configuration and scheduling result, a candidate configuration set of the Q value before training is selected, each configuration is re-scheduled and solved, and the configuration meeting the cost budget and having the minimum maximum completion time is selected as the final resource configuration.

8. A FMS dispatch and resource configuration joint optimization system, characterized in that, It includes: The model establishing module is used for establishing a Petri net model of the FMS based on the actual production relationship of the flexible manufacturing system (FMS) and using the S 3 PR net to construct the Petri net model of the FMS, and obtaining the S 3 PR net model with the warehouse delay. 3 The S 3 PR net model with the warehouse delay characterizes the workpiece set, the process sequence, the resource set and the time characteristics of each process in the FMS, and the resource set includes machining machines and transport trolley robots. The scheduling decision module is used for combining the S 3 The PR network model, based on an initial or current resource configuration scheme, executes an improved Dyna-Q algorithm under a machine time-varying environment to obtain a maximum completion time set under multiple groups of time samples; the scheduling decision module comprises scheduling strategy learning and scheduling strategy application; the scheduling strategy learning comprises an outer loop and an inner loop; the outer loop randomly samples a processing time interval and a transportation time interval of each process to obtain time samples; the inner loop completes scheduling strategy learning based on the time samples through the improved Dyna-Q algorithm, and then delivers scheduling decision results corresponding to each group of time samples to the resource configuration decision module through the scheduling strategy application. A resource configuration decision module is configured to analyze and evaluate a scheduling decision result set under multiple time samples, calculate a reward function value, explore machine number configuration under a cost constraint, generate a new resource configuration scheme, and feed back to the scheduling decision module; A scheduling and resource configuration joint optimization module is configured to repeatedly schedule the scheduling strategy learning and scheduling strategy application of the scheduling decision module based on the new resource configuration scheme, obtain a new scheduling decision result set under multiple time samples, and repeatedly start the resource configuration decision module based on the new scheduling decision result set under multiple time samples. The joint optimization iteration is iterated until a preset termination condition is met; the target of the joint optimization iteration is to minimize the system maximum completion time under the premise of meeting the resource and cost constraints; A production scheduling scheme generation module is configured to output a final resource configuration and a scheduling result after the joint optimization iteration is completed, sort a time axis according to the start time of each workpiece and each process, and generate an executable production scheduling scheme.

9. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the FMS scheduling and resource configuration joint optimization method in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the FMS scheduling and resource configuration joint optimization method in any one of claims 1-7.

Images