Learning type heterogeneous hybrid flow shop scheduling method considering worker fatigue

Abstract

The invention discloses a learning type heterogeneous hybrid flow shop scheduling method considering worker fatigue. The method comprises the following steps: S1, designing a diversified initial population generation strategy with reconstruction; s2, designing a global search mechanism, performing global search on the whole population through a tournament selection mode, combining a newly obtained population with an old population, and selecting a better half of individuals as a new population through non-dominated sorting, thereby improving the ability of the algorithm to jump out of local optimum; and S3, designing four local search strategies, using DQN to help individuals in the new population at the Pareto leading edge to select a local search strategy more suitable for the current individual state, and effectively improving the convergence speed of the algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed production scheduling in manufacturing, and in particular to a learning-based heterogeneous hybrid flow shop scheduling method that takes worker fatigue into consideration. Background Art

[0002] With the development of automation technology, industrial manufacturing capabilities are rapidly increasing in the modern world. The advancement of global trade has led to a significant increase in the number of business orders, posing significant challenges to businesses. Consequently, distributed manufacturing is becoming increasingly prevalent, with businesses transitioning from single-factory manufacturing to multi-factory collaborative production. This type of workshop structure can effectively improve the efficiency of the entire production line while effectively balancing the utilization of each machine and increasing production capacity. In actual production, due to differences in machine type, number of machines, and production environments, there are certain differences between different factories. This type of problem is known as the Distributed Heterogeneous Hybrid Flow Shop Scheduling Problem (DHHFSP). DHHFSP is a scheduling study that combines specific manufacturing processes and workshop types, and is a key technology for ensuring the efficient and stable operation of distributed manufacturing.

[0003] Establishing a reasonable distributed production scheduling solution can shorten production time, ensure product delivery deadlines, and thus improve efficiency and reduce costs. However, there is a clear correlation between workers and traditional production workshops. For example, when assigning workpieces, it is necessary to consider the availability of workers, or the impact of workers' work efficiency on production rates when operating machines to process workpieces. Under the concept of Industry 5.0, this correlation is given even greater prominence. Compared to Industry 4.0, which focuses on technological upgrades, Industry 5.0 forces us to re-examine the human dimension of production, opening up new avenues for research on distributed manufacturing and production scheduling optimization. It makes us realize that only by balancing technology and human resources can sustainable and high-quality development of the manufacturing industry be achieved.

[0004] Due to its central role in the intelligent manufacturing industry, it has successfully attracted research from scholars across numerous fields. Numerous scholars have consistently demonstrated that the machine expansion problem in hybrid flow shop scheduling often exhibits NP-hard properties. In-depth research has been conducted on flow shop scheduling using traditional mathematical operations research methods, heuristic algorithms, and intelligent optimization algorithms. Numerous new scheduling strategies have been proposed in both theoretical and applied aspects, and are widely used in a variety of fields, including automotive manufacturing, textiles, steel, and semiconductors. However, due to the high dimensionality of hybrid flow shop scheduling, the search space for the problem grows exponentially, especially as the number of workpieces increases. This can lead to algorithms experiencing long computational times and inefficient search efficiency during global searches. Consequently, approximate methods are often employed to solve shop scheduling problems.

[0005] Heuristic, metaheuristic, and reinforcement learning algorithms are widely used for large-scale scheduling problems. Heuristic algorithms rely on simple, intuitive rules to generate scheduling solutions quickly. These algorithms are based on specific priorities, such as prioritizing workpieces with shorter processing times. This approach can quickly yield feasible solutions, but due to their relatively limited decision-making basis and lack of comprehensive exploration of the entire solution space, the quality of the resulting scheduling solutions varies, making it difficult to guarantee near-optimal solutions. Metaheuristic algorithms, on the other hand, use unique search strategies to deeply explore different regions of the solution space within an acceptable timeframe, ultimately obtaining high-quality, near-optimal solutions. These algorithms offer a good balance between solution quality and computational time when solving large-scale scheduling problems. In recent years, reinforcement learning has also been increasingly applied to scheduling problems. Reinforcement learning involves an intelligent agent interacting with its environment through trial and error to learn optimal policies. Compared to traditional algorithms, reinforcement learning can dynamically adjust scheduling policies based on the real-time state of the shop floor, making it particularly well-suited for dynamically changing production environments. However, reinforcement learning faces challenges such as long training time and difficulty handling large-scale states and workspaces.

[0006] The convergence of an algorithm depends primarily on three key factors: first, whether a high-quality initial population can be obtained during algorithm initialization, which provides a good starting point for subsequent optimization; second, the design of the perturbation operator, which determines the algorithm's ability to escape local optima during the search process and help explore a wider solution space; and finally, the effectiveness of the local search strategy, which finely optimizes the current solution and improves its accuracy. These three factors work together to determine whether the algorithm can quickly and stably converge to the global optimal solution or a near-optimal solution. This study designed local search strategies based on the characteristics of the DHHFSP problem with worker factors and different levels of encoding. Search strategies for different encodings have different levels of search capabilities in the search space. This algorithm can further improve its convergence speed and search range by selecting a search strategy that is more appropriate for the current solution during the search process. The Deep Q-Network (DQN) algorithm is a reinforcement learning algorithm. Its application in shop floor scheduling is to learn the maximum cumulative reward obtained by executing actions in each state through continuous exploration and exploitation in the state-action space. The core of this algorithm is to continuously update the Q-value function to optimize the decision-making process, ensure that the action selected in a specific state can maximize the reward in the long term, and ultimately find the optimal scheduling strategy. Figure 1 It is a Gantt chart for a distributed heterogeneous hybrid flow shop scheduling problem.

[0007] NSGA-II (Non-dominated Sorting Algorithm II) is an efficient multi-objective optimization algorithm. Its core mechanism balances convergence and diversity of solutions during evolution through the synergistic effects of non-dominated sorting, crowding calculation, and an elite retention strategy. During the iterative process, the algorithm first stratifies the population using non-dominated sorting, assigning mutually non-dominated individuals to the same non-dominated layer, thus forming a hierarchical Pareto front structure. The density of individuals in the same layer is then assessed using crowding distance calculation to quantify the sparse distribution of solutions in local regions, ensuring that individuals in sparse regions are preferentially retained during environmental selection. The algorithm uses an elite retention strategy that merges parent and child populations, using the non-dominated sorting hierarchy as the primary selection criterion and the crowding distance as a secondary comparison criterion to guide the population toward the Pareto front. By maintaining a wide distribution of solutions, the algorithm effectively avoids falling into local optima. Ultimately, it generates a near-optimal solution set that evenly covers the true Pareto front, providing high-quality solutions for multi-objective optimization problems that balance global optimality and decision diversity. Summary of the Invention

[0008] In response to the above technical problems, the present invention provides a learning-based heterogeneous hybrid flow shop scheduling method that takes worker fatigue into consideration.

[0009] In order to achieve the above object, the technical solution of the present invention is as follows:

[0010] A learning-based heterogeneous hybrid flow shop scheduling method considering worker fatigue includes the following steps:

[0011] S1. Design a diverse initial population generation strategy with reconstruction;

[0012] S2. Design a global search mechanism to perform a global search on the entire population through tournament selection. The newly obtained population is merged with the old population, and the better half of the individuals are selected as the new population through non-dominated sorting, which improves the algorithm's ability to escape local optimality.

[0013] S3. Design four local search strategies and use DQN to help individuals on the Pareto frontier in the new population choose a local search strategy that is more suitable for the current individual state, effectively improving the convergence speed of the algorithm.

[0014] Among them, the step S1 is specifically as follows: first, for three different optimization objectives, three corresponding elite solutions are generated using the NEH algorithm, and other solutions in the initial population are generated by random initialization to ensure the diversity of the population. Then, some solutions are randomly selected from the randomly generated solutions, and the three elite solutions are learned and reconstructed.

[0015] Among them, step S2 specifically includes first performing tournament selection based on non-dominated hierarchy and crowding, screening out parent individuals with high hierarchy and sparse distribution to ensure population diversity, then exchanging gene fragments of the selected parents through crossover operation to generate offspring to explore the solution space, and performing mutation operation on the offspring with a certain probability to prevent the algorithm from falling into local optimality, then merging the parents and offspring into a double-sized population, re-performing non-dominated sorting and crowding calculation, and finally screening out the top 50% of solutions as the next generation parents through the elite retention strategy.

[0016] Specifically, step S3 proposes a local search strategy for four-layer encoding to accelerate the convergence of the population, and uses DQN to assist in optimizing the non-dominated solution. The optimal strategy is dynamically selected according to the current state of the non-dominated solution. After executing the action to generate a new solution, the reward is distributed by comparing the dominance relationship between the new and old solutions. The state transfer process, i.e., the current state, action, reward, and new state, is then stored in the experience pool. When the sample accumulation exceeds the threshold, the network is trained by randomly sampling batch data, and the parameters are updated to optimize the strategy selection ability.

[0017] The beneficial effects of the present invention are:

[0018] 1. The present invention takes minimizing the maximum completion time, minimizing machine energy consumption and minimizing worker idle time as optimization goals, and solves the distributed heterogeneous hybrid flow shop scheduling problem considering worker fatigue.

[0019] 2. The present invention introduces multiple local search operators tailored to the characteristics of the problem into the non-dominated sorting algorithm to improve the efficiency of the local search, thereby accelerating the overall convergence speed of the algorithm and adaptively selecting the local search strategy throughout the search process.

[0020] 3. The present invention adopts a random generation method to effectively improve the diversity of the population, and adopts the elite solution generated by NEH and the reconstructed partial solution to effectively improve the quality of the initial population.

[0021] 4. The DQN algorithm of the present invention is used to dynamically select search strategies. Based on the current state of the solution and the historical feedback information of each local search strategy, an appropriate strategy is selected to further improve the quality of non-dominated solutions.

[0022] 5. The present invention has simple logic, is easy to implement and expand, and can extend the optimizer to meet most scheduling problems in the current intelligent manufacturing production field. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a Gantt chart for the distributed heterogeneous hybrid flow shop scheduling problem.

[0024] Figure 2 This is a diagram of the evolution process of DQN-assisted non-dominated solutions.

[0025] Figure 3 It is a schematic diagram of initialization and reconstruction.

[0026] Figure 4 Schematic diagram of four local search strategies.

[0027] Figure 5 It is the algorithm flow chart.

[0028] Figure 6 This is the Pareto front graph obtained by DQCNSGA-II and the other seven comparison algorithms. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0030] A learning-based heterogeneous hybrid flow shop scheduling method considering worker fatigue includes the following steps:

[0031] S1. Design a diverse initial population generation strategy with reconstruction;

[0032] S2. Design a global search mechanism to perform a global search on the entire population through tournament selection. The newly obtained population is merged with the old population, and the better half of the individuals are selected as the new population through non-dominated sorting, which improves the algorithm's ability to escape local optimality.

[0033] S3, such as Figure 2 As shown in the figure, four local search strategies are designed, and DQN is used to help individuals on the Pareto frontier in the new population choose a local search strategy that is more suitable for the current individual state, which effectively improves the convergence speed of the algorithm.

[0034] Among them, such as Figure 3 As shown, the step S1 is specifically as follows: first, for three different optimization objectives, three corresponding elite solutions are generated using the NEH algorithm, and other solutions in the initial population are generated by random initialization to ensure the diversity of the population. Then, some solutions are randomly selected from the randomly generated solutions, and the three elite solutions are learned and reconstructed.

[0035] Among them, step S2 specifically includes first performing tournament selection based on non-dominated hierarchy and crowding, screening out parent individuals with high hierarchy and sparse distribution to ensure population diversity, then exchanging gene fragments of the selected parents through crossover operation to generate offspring to explore the solution space, and performing mutation operation on the offspring with a certain probability to prevent the algorithm from falling into local optimality, then merging the parents and offspring into a double-sized population, re-performing non-dominated sorting and crowding calculation, and finally screening out the top 50% of solutions as the next generation parents through the elite retention strategy.

[0036] Among them, such as Figure 4 As shown, step S3 is specifically as follows: in order to accelerate the convergence speed of the population, a local search strategy for four-layer encoding is proposed, and DQN is used to assist in optimizing the non-dominated solution. According to the current state of the non-dominated solution, the optimal strategy is dynamically selected. After the action is executed to generate a new solution, the reward is distributed by comparing the dominance relationship between the new and old solutions. Then, the state transfer process, that is, the current state, action, reward, and new state, is stored in the experience pool. When the sample accumulation exceeds the threshold, the network is trained by randomly sampling batch data, and the parameters are updated to optimize the strategy selection ability.

[0037] When implementing:

[0038] 1. Population initialization module

[0039] Reasonable initialization can effectively improve the quality of the initial population. Specifically, the NEH algorithm is first used to generate three elite solutions for each of the three objectives. Subsequently, the remaining solutions in the population are randomly generated and used as common solutions. For each of the three objectives, some common solutions are randomly selected and reconstructed by learning the three elite solutions. Regarding the specific solution learning and reconstruction strategy, the pseudo code is shown in Algorithm 1.

[0040]

[0041] 2. Improved module based on DQN

[0042] In order to further improve the quality of the solution, four sequence-related local search operations are used, corresponding to the four layers of coding. The specific operation process is as follows:

[0043] 1) LS1 (Critical_Job_Swap): First, determine the critical and non-critical job in the current solution (RS) and the best solution (BS). Then, randomly select a critical job in BS. If the positions of the selected job in RS and BS are different, the jobs are swapped. If the positions are the same, a critical job in RS is randomly swapped with a non-critical job.

[0044] 2) LS2 (Factory_Mutation): Calculate the total load of each factory, find the factories with the largest and smallest loads, randomly select a workpiece assigned to the factory with the largest load and reallocate it to the factory with the smallest load, and finally return a new chromosome list containing this change for use in the subsequent optimization process.

[0045] 3) LS3 (Worker_Mutation): To avoid low efficiency caused by continuous processing by workers, traverse the worker processing sequence. If three consecutive workpieces in the worker sequence are assigned to the same worker, the third workpiece is randomly reallocated to another worker. Otherwise, two workpieces are randomly selected from two workers to exchange processing workers.

[0046] 4) LS4 (Speed_Mutation): This process traverses all machines in each factory, calculates the running time and waiting time of each machine, records the machines with the longest and shortest waiting times, and reduces the speed of the machines with the longest waiting times to reduce their waiting time. It also increases the speed of the machines with the shortest waiting times to balance the machine load.

[0047] A greedy strategy and random exploration strategy are used to select actions. If the generated random number is less than ε, the evaluation network is used to calculate the value of each action. After processing, the action with the maximum value is selected. If there are multiple maximum values, an action is randomly selected. Otherwise, an action is randomly selected and finally returned to the selected action for reinforcement learning. This approach not only maximizes the reward by leveraging the known optimal action, but also discovers potentially better strategies by exploring new actions. This effectively balances exploration and exploitation, improving the performance and generalization of the reinforcement learning model. The reward value is updated as follows:

[0048]

[0049] where n l Indicates the amount by which the new solution improves on the target value compared to the old solution, n e Indicates the number of new solutions that have the same fitness value as the old solution, n m Indicates the amount by which the new solution deteriorates compared to the old solution in terms of target value. The pseudo code of the DQN-based local search algorithm is shown in Algorithm 2:

[0050]

[0051] 3. System Process

[0052] The algorithm consists of three main parts: a diversified generation strategy to generate the initial population, a global search mechanism, and a local search based on reinforcement learning. In the initialization phase, a combination of the NEH heuristic algorithm and randomization is used to generate a diverse and high-quality initial population, and a reconstruction strategy is used to further improve the population quality. In terms of global search, tournament selection is used to help the population complete crossover and mutation operations. Finally, reinforcement learning DQN is used to assist the population in local search operations. The algorithm flow chart is as follows: Figure 5 As shown, the algorithm pseudo code is as follows:

[0053]

[0054] In addition, to verify the effectiveness of the algorithm, we conducted experiments with its DQCNSGA-II algorithm and seven other algorithms. Figure 6 This is the Pareto front graph obtained by 8 algorithms on instances of different sizes. It can be seen that the Pareto front obtained by DQCNSGA-II is better than that of other algorithms.

[0055] It should be understood that the above-described specific embodiments of the present invention are merely illustrative or illustrative of the principles of the present invention and do not constitute limitations of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included within the scope of protection of the present invention. In addition, the appended claims are intended to cover all variations and modifications that fall within the scope and metes and bounds of the appended claims, or equivalents thereof.

Claims

1. A learning-based heterogeneous hybrid flow shop scheduling method considering worker fatigue, characterized by: The following steps are involved: S1. Design a diverse initial population generation strategy with reconstruction; S2. Design a global search mechanism to perform a global search on the entire population through tournament selection. The newly obtained population is merged with the old population, and the better half of the individuals are selected as the new population through non-dominated sorting, which improves the algorithm's ability to escape local optimality. S3. Design four local search strategies and use DQN to help individuals on the Pareto frontier in the new population choose a local search strategy that is more suitable for the current individual state, effectively improving the convergence speed of the algorithm.

2. The learning-based heterogeneous hybrid flow shop scheduling method considering worker fatigue according to claim 1 is characterized by: The step S1 specifically comprises the following steps: first, for three different optimization objectives, three corresponding elite solutions are generated using the NEH algorithm, and other solutions in the initial population are generated by random initialization to ensure the diversity of the population; then, some solutions are randomly selected from the randomly generated solutions, and the three elite solutions are learned and reconstructed.

3. The learning-based heterogeneous hybrid flow shop scheduling method considering worker fatigue according to claim 1 is characterized by: The step S2 specifically comprises the following steps: firstly, a tournament selection is performed based on the non-dominated hierarchy and crowding degree to screen out parent individuals with high hierarchy and sparse distribution to ensure population diversity; then, a crossover operation is performed on the selected parent generation to exchange gene fragments to generate offspring to explore the solution space; and a mutation operation is performed on the offspring with a certain probability to prevent the algorithm from falling into a local optimum; then, the parent generation and offspring are merged into a double-sized population, and the non-dominated sorting and crowding degree calculation are re-performed; finally, an elite retention strategy is used to screen out the top 50% of solutions as the next generation of parents.

4. The learning-based heterogeneous hybrid flow shop scheduling method considering worker fatigue according to claim 1 is characterized by: Specifically, step S3 proposes a local search strategy for four-layer encoding to accelerate the convergence of the population, and uses DQN to assist in optimizing the non-dominated solution. The optimal strategy is dynamically selected according to the current state of the non-dominated solution. After executing the action to generate a new solution, the reward is distributed by comparing the dominance relationship between the new and old solutions. The state transfer process, i.e., the current state, action, reward, and new state, is then stored in the experience pool. When the sample accumulation exceeds the threshold, the network is trained by randomly sampling batch data, and the parameters are updated to optimize the strategy selection ability.

Images