A distributed congestion flow water-works scheduling optimization system based on reinforcement learning

Abstract

The application belongs to the field of distributed production scheduling in manufacturing industry, and particularly relates to a distributed blocked flow shop scheduling optimization system based on reinforcement learning, which comprises a scheduling sequence diversification initialization module, an improved module based on Q-learning and a local search module based on neighborhood reconstruction; the scheduling sequence diversification initialization module designs a diversified initial population generation strategy, the improved module based on Q-learning designs a global search mechanism based on a reinforcement learning mechanism, and a search operator is adaptively selected according to a search state and historical experience of the operator; and the local search module based on neighborhood reconstruction comprises a deep local search strategy based on neighborhood reconstruction and an improved strategy based on path reconnection. The application has simple logic, is easy to implement and easy to expand, and can expand the optimizer to most scheduling problems in the current intelligent manufacturing production field.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of distributed production scheduling in manufacturing industry, and particularly relates to a distributed blocking flowshop scheduling optimization system based on reinforcement learning. BACKGROUND

[0002] In the prior art, distributed production environments are becoming increasingly important as they provide enterprises with higher quality, shorter delivery times and lower cost products, from the original single factory manufacturing to multi-factory collaborative production. Production scheduling is a component of manufacturing systems, which directly affects the efficiency and competitiveness of enterprises. Especially the distributed blocking flowshop scheduling problem, which has important application in the manufacturing process of enterprises, has attracted the attention of many researchers. The distributed blocking flowshop scheduling problem (DBFSP) is an extension of the traditional blocking flowshop scheduling problem, and its core is to allocate workpieces to different factories and allocate production resources and process scheduling within the factory to ensure high production rate and resource utilization. PFSP has been proven to be an NP-hard problem. DBFSP considers the condition of no buffer between machines on the basis of distributed permutation flowshop. Therefore, DBFSP is also an NP-hard problem.

[0003] The main solution method for distributed workshop scheduling problem is approximation method. When solving complex combinatorial optimization problems by exact algorithm, due to the large scale of problem solution space and long calculation time, it is limited to the solution of small-scale problems, which is a challenging problem. For large-scale problems, heuristic, meta-heuristic and hybrid algorithms are widely used, among which the advantage of heuristic algorithm is that it can quickly obtain scheduling solution in relatively short time, but the quality of solution is usually difficult to guarantee. However, meta-heuristic algorithm can obtain high-quality approximate optimal solution within an acceptable time frame. DBFSP allows modeling in various production scheduling systems, and there is no buffer between machines in some processing stages of production manufacturing system, which means that the workpiece completed on a machine must stay on the machine before the next machine is available and block itself, and due to the strong constraints of blocking flowshop scheduling, the manufacturing completion time is greatly delayed. Therefore, it is necessary to improve productivity by minimizing machine blocking and idle time. Minimizing completion time is a key practice goal in the production industry. Under dynamic production environment, how to balance the use of resources among production enterprises, including reducing product inventory and finished product turnover rate, is considered to have practical significance. Therefore, it is necessary to expand the existing research to obtain effective scheduling method to solve the scheduling problem in production environment. The Gantt chart of DBFSP is shown in Figure 1

[0004] ​The ability of metaheuristic algorithms to obtain high-quality solutions depends largely on the design and selection of perturbation operators and local search methods. In this study, different search operators are designed according to the characteristics of the problem, and each search operator has different search capabilities in different regions of the search space. Metaheuristic can effectively balance the exploration and development capabilities of the algorithm by selecting search operators with different characteristics at different stages of the search process. During the search of the metaheuristic algorithm for the approximate optimal solution, the historical performance of the search operator needs to be considered in order to dynamically select and adopt the most suitable operator. As a branch of machine learning algorithms, Q-learning algorithms have achieved a series of research results in solving scheduling problems. Machine learning techniques serve metaheuristic algorithms by extracting knowledge rules from the data generated during the entire search process. Reasonably integrating knowledge into the search process at different stages of population evolution can guide the algorithm to make optimal decisions and effectively improve the quality of solutions.

[0005] SS uses the intelligent iteration mechanism of "dispersion-convergence aggregation" to construct high-quality and diverse solutions in the reference set, and obtains the global optimal solution or satisfactory solution of the problem through the sub-set merging method and the reference set updating method. The SS framework includes five system sub-methods, namely the diversification generation method, the solution improvement method, the reference set updating method, the sub-set generation method and the solution combination method. The entire framework is very flexible, and each sub-method in the framework can be implemented and replaced with different methods according to the complexity or different requirements of the actual problem. The SS algorithm is used to solve various engineering optimization problems, and the solving process is simple. The SS algorithm has certain research foundation and advantages in solving the distributed blocking flow shop scheduling problem. SUMMARY

[0006] The application provides a distributed blocking flow shop scheduling optimization system based on reinforcement learning, which minimizes the maximum completion time as an optimization objective, solves the distributed blocking flow shop scheduling problem, and can optimize the operation efficiency and performance of the distributed blocking flow shop scheduling system.

[0007] In order to achieve the above purpose, the application provides the following technical scheme:

[0008] A distributed blocking flow shop scheduling optimization system based on reinforcement learning, characterized by comprising a scheduling sequence diversification initialization module, a Q-learning-based improvement module, and a local search module based on neighborhood reconstruction.

[0009] Preferably, the scheduling sequence diversification initialization module designs a diversification initial population generation strategy, first, an initial sequence is randomly generated to ensure the diversity of the initial sequence; then, the first f (f is the number of factories) jobs are taken out in turn and placed in the first position of each factory, and the remaining jobs are taken out in turn and inserted into the best position in all factories, and the above process is repeated until a certain number of individuals are generated to form a population. All subsequent operations are performed on individuals in the population.

[0010] Specifically, first, the first processing job of each factory is determined, and the first f jobs after random sorting are inserted into the first position of each factory; second, the job to be processed on the last machine of each factory is selected, and all remaining jobs are arranged in ascending order according to the processing time of the job on the last machine, and the first f jobs after sorting are inserted into the last position of each factory; each factory performs NEH heuristic-based insertion operation, and all remaining jobs perform NEH-based neighborhood insertion search except the first and last jobs of each factory. This makes the system obtain higher precision results when solving distributed blocking flow shop scheduling problems.

[0011] Preferably, the improved module based on Q-learning designs a global search mechanism based on reinforcement learning mechanism, which adaptively selects search operators according to the history experience of search state and operator. Compared with traditional single search operator selection, adaptive selection of search operators according to real-time state of individuals can perform exploration of different depths, improving the ability of the algorithm to escape from local optimum.

[0012] Specifically, five neighborhood knowledge-based neighborhood disturbance strategies are designed, and the neighborhood disturbance operators in the strategy set have two types, the first type is based on the operation within the key factory (C fac ), and the other type is the operation between the key factory and other factories; the Q-learning algorithm selects the most suitable disturbance strategy at each step in the Refset update process; the action set (A=N1, N2, …, N5) is defined as a neighborhood search operator with five different disturbance intensities. The combination model of KCLSS and reinforcement learning (RL) is shown in Figure 2 , and the learning module is shown in Figure 3 . The last neighborhood disturbance mechanism is a job block-based search operator, which aims to increase the disturbance intensity. The five neighborhood structures are shown in Figure 4 .

[0013] Specifically, in order to optimize the job sequence, five sequence-related operations are adopted; n represents the number of jobs, m represents the number of machines; Rand fac is a random factory, C fac is a key factory;

[0014] The specific operation process is described as follows:

[0015] N1 (Insert_Rand fac _C fac ) : Perform the insertion operation between C fac and the randomly selected factory Rand fac ; first, a non-key factory is randomly selected, then each workpiece in C fac is taken out in turn and reinserted into all positions in Rand fac , and the position producing the best C max is recorded;

[0016] N2 (Swap_Rand fac _C fac ) : Perform the exchange operation between C fac and the randomly selected factory Rand fac ; first, a non-key factory is randomly selected, then each workpiece in C fac is taken out in turn and exchanged with each position in Rand fac , and the position producing the best C max is recorded;

[0017] N3 (Insert_C fac ) : Randomly select a workpiece in C fac and insert it into all possible positions in the factory, if the C fac of C max after insertion is better than the current solution, then retain the new solution;

[0018] N4 (Swap_C fac ) : Randomly select two workpieces in C fac , then exchange them, if C max is improved after the exchange, then accept the new solution;

[0019] N5 (Insert_block_C fac ) : First, randomly select a starting position in C fac , extract a workpiece block consisting of η (η = rand[2, 4]) consecutive workpieces and remove it from the original position, then reinsert the workpiece block into the best position in C fac .

[0020] An improved ε-greedy strategy is adopted, in the early stage of population iteration evolution, a higher probability is used to explore new behaviors, with the increase of training time, the agent turns from exploring new actions to selecting the action that can obtain the maximum reward at present, the advantage of using this strategy is to ensure that the agent can explore a wider search area, while retaining a certain depth of exploration ability; The improved strategy is specifically described as follows:

[0021]

[0022]

[0023]

[0024] Where a * represents the action with the maximum Q value in s t , A(s t ) represents the set of all available actions in s t , P rand is a sample value subject to standard normal distribution, and T max represents the stop criterion (T max =10×n×m×f).

[0025] Preferably, the local search module based on neighborhood reconstruction is a deep local search strategy based on path reconnection. Path reconnection strategy is an effective search method to explore the neighborhood of two given solutions, which is applied in this stage to generate a new valuable solution. Here, the current best individual Cbest is compared with the current individual Ci. Insertion or exchange transformation is performed on the current individual Ci to Cbest, and an intermediate individual is generated after each operation operation. Finally, all intermediate individuals are evaluated, and the optimal individual is retained.

[0026] Specifically, the deep local search strategy based on neighborhood reconstruction is proposed to enhance the local optimization performance, a neighborhood reconstruction strategy based on path reconnection is proposed, and reconstruction operation is performed on the current individual and the optimal individual in the current population, for each workpiece in the key factory, it is taken out in turn and inserted into all possible positions of the factory, and the neighborhood structure is reconstructed as shown in Figure 5 .

[0027] Specifically, the local search method of neighborhood reconstruction is a commonly used method for solving flow shop scheduling problems, and the basic idea is to search for a better solution in the neighborhood of the current solution until no better solution can be found; first, select the best individual in the current population, and then take out each workpiece in turn, then convert to the current optimal solution through the insertion operation to generate the current neighborhood solution, search for a better solution in the current neighborhood solution, that is, find the neighborhood solution that minimizes the objective function value. If a better solution is found, update the current solution to the neighborhood solution, repeat the above steps until the search is complete.

[0028] The present application has the following beneficial effects:

[0029] (1) The present application introduces multiple neighborhood disturbance operators in the distributed search algorithm. The neighborhood structure is designed according to the characteristics of the problem to improve the efficiency of local search. The neighborhood disturbance strategy is adaptively selected during the entire search process.

[0030] (2) The Q-learning algorithm is used to dynamically select the search strategy. Based on the search state and the historical feedback information of each disturbance operator, the appropriate disturbance operator is selected to further balance the exploration and development capabilities.

[0031] (3) The reconstructed neighborhood structure is designed to enhance the local search capability. The synergistic evolution between SS sub-methods can effectively improve the quality of the population.

[0032] (4) The present application has simple logic, easy implementation and easy expansion, and can extend the optimizer to meet most scheduling problems in the current intelligent manufacturing production field. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the examples of the present application, some of the drawings in the examples of the present application will be briefly introduced below.

[0034] Figure 1 is a Gantt chart of the distributed blocking flow shop scheduling problem in the prior art.

[0035] Figure 2 is a combination model diagram of KCLSS and RL in the present application.

[0036] Figure 3 is a learning module principle diagram in the present application.

[0037] Figure 4 is a schematic diagram of five neighborhood structures in the present application.

[0038] Figure 5 is a neighborhood reconstruction schematic diagram in the present application.

[0039] Figure 6 is an algorithm flowchart in the present application. DETAILED DESCRIPTION

[0040] In order to more clearly illustrate the purpose, technical solutions and advantages of the present application, the functions and advantages of each module are specifically explained below with reference to the drawings.

[0041] Embodiment one

[0042] 1. Scheduling sequence diversification initialization module

[0043] The construction heuristic usually obtains a heuristic solution according to the characteristics of the problem. First, an initialization sequence is randomly generated Then, the first f jobs are taken out one by one and arranged in the first position of each factory. The remaining jobs are taken out in turn, and all possible positions of all factories are tested, the best position is selected for insertion, and the processing load of each factory cannot exceed the average processing load ). The pseudo code of the population generation strategy is shown in Algorithm 1.

[0044]

[0045]

[0046] 2. Improved module based on Q-learning

[0047] In order to optimize the job sequence, five sequence-related operations are adopted. n represents the number of jobs, and m represents the number of machines. Rand fac is a random factory, and C fac is a critical factory. The operation process is described as follows.

[0048] The specific description is as follows:

[0049] 1) N1 (Insert_Rand fac _C fac ): Perform the insert operation between C fac and the randomly selected factory Rand fac . First, a non-critical factory is randomly selected, then each job in C fac is taken out in turn and reinserted into all positions in Rand fac , and the position producing the best C max is recorded.

[0050] 2) N2 (Swap_Rand fac _C fac ): Perform the swap operation between C fac and the randomly selected factory Rand fac .The exchange operation is performed between two randomly selected non-key factories. First, a non-key factory is randomly selected, then two jobs are randomly selected from the factory and swapped. If the C fac value is improved, the new solution is accepted. fac max

[0051] 3) N3 (Insert_C fac ): A job is randomly selected from C fac and inserted into all possible positions in the factory. If the C fac value is improved after insertion, the new solution is accepted. max

[0052] 4) N4 (Swap_C fac ): Two jobs are randomly selected from C fac and swapped. If the C max value is improved after swapping, the new solution is accepted.

[0053] 5) N5 (Insert_block_C fac ): First, a starting position is randomly selected from C fac , a block of η (η = rand[2, 4]) consecutive jobs is extracted and removed from the original position, and then the block is inserted into the optimal position in C fac .

[0054] An improved ε-greedy strategy is adopted, which explores new behaviors with a higher probability in the early stages of population iteration evolution. As the training time increases, the agent shifts from exploring new actions to selecting actions that can currently obtain the maximum reward. The advantage of this strategy is to ensure that the agent can explore a wider search area while retaining a certain depth of exploration ability. The improved strategy is specifically described as follows:

[0055]

[0056]

[0057]

[0058] where a * represents the action with the maximum Q value in s t , A(s t ) represents the set of all available actions in s t , P rand is a sample value subject to a standard normal distribution, and T max represents the stopping criterion (T max = 10 × n × m × f). ​​​

[0059] According to the workpiece processing population sequence generated by the scheduling sequence diversification initialization module, a search operator is dynamically selected according to the Q-learning algorithm feedback information. First, the agent observes the current environment, and according to the state s t at time t, an action a t is taken from the set of available actions. Then, the agent interacts with the environment. After the action a t is performed, the state of the agent is switched to s t+1 , and the agent obtains a reward or punishment r t from the environment. Sequence-related operations with different disturbance intensities are used to adjust the sequence arrangement and find a better workpiece processing arrangement order.

[0060] 3. Neighborhood reconstruction local search module based on path reconnection

[0061] The neighborhood reconstruction local search method is a common method for solving flow shop scheduling problems. The basic idea is to search for a better solution in the neighborhood of the current solution until a better solution cannot be found. First, the current optimal individual and the current individual are selected, and then each workpiece of the current individual is taken out. Then, the current optimal solution is converted through the insertion operation, and a better solution is searched in the current neighborhood solution, that is, a neighborhood solution that minimizes the objective function value is found. If a better solution is found, the current solution is updated to the neighborhood solution, and the above steps are repeated until the search is completed.

[0062] 4. System flow

[0063] The KCLSS algorithm mainly consists of three parts: a diversification generation strategy to generate an initial population, a global search mechanism based on reinforcement learning mechanism, and a neighborhood reconstruction deep local search mechanism based on path reconnection. In the initialization stage, a method combining construction heuristics and randomization is used to generate a high-quality initial population. In terms of global search, a neighborhood disturbance strategy pool with different structural characteristics is designed, and combined with the Q-learning algorithm, a suitable disturbance strategy is dynamically selected. The neighborhood reconstruction strategy guides the individual to search in the potential advantage area of the solution space. The algorithm pseudocode is shown in Algorithm 2, and the algorithm flowchart is shown in Figure 6 .

[0064]

[0065] The above describes the basic principles, main features and advantages of the examples of the present application. Related practitioners should understand that the present application is not limited by the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. The present application can be applied to any other field with optimized properties. The present application will also have various changes and improvements, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A distributed congestion flow-shop scheduling optimization system based on reinforcement learning, characterized in that: The scheduling sequence diversification initialization module, the improved module based on Q-learning, and the local search module based on neighborhood reconstruction are included. The scheduling sequence diversification initialization module designs a diversified initial population generation strategy. Firstly, an initial sequence is randomly generated to ensure the diversity of the initial sequence. Then, the first f jobs are sequentially taken out and placed in the first position of each factory, and the remaining jobs are sequentially taken out and inserted into the best position in all factories. The above process is repeated until a certain number of individuals are generated to form a population. Specifically, firstly, the first processing job of each factory is determined, and the first f jobs after random sorting are sequentially inserted into the first position of each factory. Secondly, the job to be processed on the last machine of each factory is selected, and all remaining jobs are arranged in ascending order according to the processing time of the jobs on the last machine. The first f jobs after sorting are inserted into the last position of each factory. Each factory performs an insertion operation based on the NEH heuristic, and all remaining jobs perform a neighborhood insertion search based on the NEH, except for the first and last jobs of each factory. This enables the system to obtain higher precision results when solving distributed blocking flow shop scheduling problems. The improved module based on Q-learning designs a global search mechanism based on reinforcement learning mechanism, which adaptively selects search operators according to search state and historical experience of operators; five strategy sets based on neighborhood disturbance with neighborhood knowledge are designed, and the neighborhood disturbance operators in the strategy sets have two types, the first type is based on the operation inside the key factory (C fac ), and the other type is the operation between the key factory and other factories; Q-learning algorithm selects the most appropriate disturbance strategy at each step in the Refset update process; the action set (A=N1, N2, …, N5) is defined as the neighborhood search operator with five different disturbance intensities; in order to optimize the workpiece sequence, five sequence-related operations are adopted; n represents the number of workpieces, m represents the number of machines; Rand fac is a random factory, and C fac is a key factory; The specific operation process is described as follows: N1(Insert_Rand fac _C fac ):In C fac and randomly selected factory Rand fac between the implementation of the insertion operation; first, a random selection of a non-key factory, and then, in turn, take each workpiece in C fac re-inserted into all positions Rand fac , record the position of the best C max ; N2 (Swap_Rand fac _C fac ) : perform swap operation between C fac and randomly selected factory Rand fac ; first, randomly select a non-critical factory, then, take out each workpiece in C fac and each position of workpiece in Rand fac in turn, exchange, record the position that produces the best C max ; N3(Insert_C fac ): In C fac Randomly select a workpiece and insert it into all possible positions in the factory. If after insertion, C fac C max If the new solution is better than the current solution, then retain the new solution; N4 (Swap_C fac ) : Choose two jobs at random in C fac , then swap them, if C max is improved after the swap, accept the new solution; N5 (Insert_block_C fac ) : First, a starting position is randomly selected in C fac , a block of η (η = rand[2,4]) consecutive jobs is extracted and removed from the original position, and then the block is reinserted in the best position in C fac . An improved ε-greedy strategy is used. In the early stage of population iteration evolution, a higher probability is used to explore new behaviors. As the training time increases, the agent shifts from exploring new actions to selecting actions that can currently obtain the maximum reward. The advantage of this strategy is to ensure that the agent can explore a wider search area while retaining a certain depth of exploration capability. The improved strategy is specifically described as follows: where a * represents the maximum Q value in the s t state, A(s t ) represents a set of all possible actions in the s t state, P rand is a sample value subject to a standard normal distribution, and T max represents a stop criterion, T max = 10 x n x m x f. The local search module based on neighborhood reconstruction is a deep local search strategy based on path reconnection. Path reconnection strategy is an effective search method to explore the neighborhood of two given solutions. It is applied in this stage to generate a new valuable solution. Firstly, the current best individual Cbest is compared with the current individual Ci. Insertion or exchange conversion is performed on the current individual Ci to Cbest, and an intermediate individual is generated after each operation. Finally, all intermediate individuals are evaluated, and the optimal individual is retained. 2.The distributed congestion flow-shop scheduling optimization system based on reinforcement learning of claim 1, wherein: The deep local search strategy based on neighborhood reconstruction is proposed to enhance the local optimization performance. A neighborhood reconstruction strategy based on path reconnection is proposed. The reconstruction operation is performed on the current individual and the optimal individual in the current population. For each job in the key factory, each job is sequentially taken out and inserted into all possible positions in the factory.

3. The distributed congestion flow-shop scheduling optimization system based on reinforcement learning according to claim 2, wherein: The deep local search method based on neighborhood reconstruction is a common method for solving flow shop scheduling problems. The basic idea is to search for a better solution in the neighborhood of the current solution until no better solution can be found. Firstly, the optimal individual of the current population is selected, and each job is sequentially taken out. Then, the current neighborhood solution is generated through the insertion operation. In the current neighborhood solution, a better solution is searched, i.e., the neighborhood solution that minimizes the objective function value is found. If a better solution is found, the current solution is updated to the neighborhood solution. The above steps are repeated until the search is completed.

Images