Distributed assembly hybrid flow shop scheduling method based on super-heuristic reinforcement learning

Abstract

The application discloses a distributed assembly hybrid flow shop scheduling method based on super-heuristic reinforcement learning, and belongs to the technical field of hybrid flow shop scheduling. The method comprises the following steps: EE-DAHFSP-PM MILP model construction: according to the distributed assembly hybrid flow shop scheduling requirements of multi-objective energy saving with machine maintenance, a planning model of the distributed assembly hybrid flow shop with multi-objective energy saving and machine maintenance, namely, the EE-DAHFSP-PM MILP model, is established; the optimization target of the EE-DAHFSP-PM MILP model is to minimize the maximum completion time of the factory and minimize the total carbon emission of all factories; EE-DAHFSP-PM MILP model solution: based on the Q learning super-heuristic evolutionary algorithm QLHHEA, Q learning is used as HLS to manage and manipulate LLH to search for an optimal LLH sequence on a heuristic search space, the optimal LLH sequence found is used on the solution space to search for an optimal pareto solution set, and a scheduling scheme is obtained.

Description

TECHNICAL FIELD

[0001] The embodiment of the application relates to the technical field of mixed flow shop scheduling, and particularly relates to a distributed assembly mixed flow shop scheduling method based on hyper-heuristic reinforcement learning. BACKGROUND

[0002] Under the background of economic globalization and sustainable industrial development, green industry has become a prevailing trend of new industrialization. As a modern energy-saving and environment-friendly manufacturing mode, low-carbon manufacturing can promote the upgrading and transformation of green industry. Energy consumption leads to the release of a large amount of greenhouse gases. This situation aggravates climate change, making energy saving and emission reduction more important. In addition, with the rapid development of intelligent manufacturing and low-carbon manufacturing, distributed manufacturing can respond to the rapidly changing market, improve resource utilization and production efficiency, and make a positive contribution to sustainable green industry. At the same time, many scholars have begun to conduct extensive and in-depth research on the distributed assembly hybrid flow shop scheduling problem (DAHFSP). DAHFSP has been proven to be a strongly NP-hard problem, and it is too challenging to implement a robust and high-quality scheduling scheme to solve this problem in a short time. Hyper-heuristic algorithm (HHA) is a kind of advanced automated algorithm proposed in recent years, which is used to solve complex computational search problems. At present, researchers have widely used hyper-heuristic methods to solve various shop scheduling problems, such as hybrid shop scheduling problem, dynamic job shop scheduling problem, flexible job shop scheduling problem, distributed assembly flow shop scheduling problem, mixed flow shop scheduling problem, etc. This makes HHA very likely to become a promising algorithm framework for solving shop scheduling problems.

[0003] The research found that the prior art has some problems: first, most DAHFSPs ignore the energy consumption problem, and low-carbon manufacturing is a goal universally pursued by modern enterprises, which requires the industry to focus on the sustainability and energy saving and emission reduction problems. Therefore, in order to ensure product quality and production efficiency, in addition to considering economic benefit indicators, energy consumption indicators also need to be further considered. Secondly, most researches are almost based on the idea that machines are always available and have a long life cycle, without considering the actual production situation. In fact, machines will wear and tear during normal use, which causes the machines to be in an unusable state for a certain period of time. This situation will disrupt the production plan, cause delivery delays, additional costs, and reduced customer satisfaction, and other adverse effects. Therefore, it is of great practical significance and potential value to study the energy-saving distributed assembly hybrid flow shop problem with machine maintenance in the actual production environment. For the energy-saving distributed assembly hybrid flow shop problem with machine maintenance (EE-DAHFSP-PM), the Q-learning-based hyper-heuristic evolutionary algorithm (QLHHEA) can provide appropriate strategies to select appropriate search behaviors and guide the search trend to promising areas to improve the solution efficiency and quality and reduce energy consumption. SUMMARY

[0004] In order to overcome the technical defects in the above-mentioned prior scheduling scheme, the embodiments of the present application provide a distributed assembly hybrid flow shop scheduling method based on hyper-heuristic reinforcement learning, which determines the factory allocation of products, the processing order of products in the factory, the processing order of workpieces to which the products belong, and the maintenance time points of each machine in the EE-DAHFSP-PM, so as to minimize the maximum completion time of all factories and minimize the carbon emissions of all factories.

[0005] The technical scheme of the scheduling method is as follows:

[0006] S1, MILP model construction of EE-DAHFSP-PM: according to the requirements of the multi-objective energy-saving distributed assembly hybrid flow shop scheduling with machine maintenance, a multi-objective energy-saving distributed assembly hybrid flow shop planning model, i.e. the MILP model of EE-DAHFSP-PM, is established;

[0007] The optimization objectives of the MILP model of EE-DAHFSP-PM are to minimize the maximum completion time of the factory, and to minimize the total carbon emissions of all factories;

[0008] S2, solving the MILP model of EE-DAHFSP-PM:

[0009] The Q-learning-based hyper-heuristic evolutionary algorithm QLHHEA uses Q-learning as HLS to manage and manipulate LLH to find the optimal LLH sequence in the heuristic search space, and then applies the found optimal LLH sequence to the solution space to search for the optimal Pareto solution set and obtain a scheduling plan.

[0010] Specifically, in S2, the specific steps of solving the MILP model of EE-DAHFSP-PM include:

[0011] S2.1. Design the encoding and decoding scheme of EE-DAHFSP-PM;

[0012] S2.2. Construct a low-level heuristic LLHs pool. The LLHs pool can be divided into two parts: products and processes. Neighborhood searches are performed on product sequences and workpiece sequences respectively.

[0013] S2.3. Initialize the parameters of the Q-learning hyper-heuristic evolutionary algorithm, which specifically include population size popsize, learning rate λ, discount rate γ, elite high-level individual ratio Initial greed rate e0, final greed rate e f ;

[0014] S2.4. After initializing the population, screen the individuals in the population using the fitness function to obtain the non-dominated solution set Π in the population;

[0015] S2.5, Destruction-Reconstruction: Perform destruction and reconstruction operations on the non-dominated solution set π, P Randomly select d products from the dataset and perform complete destruction-reconstruction on the workpiece of each product. Then, find the union of the destroyed and reconstructed population and Π, and perform non-dominated solution search again to obtain a new non-dominated solution set Π.

[0016] S2.6, energy-saving operation: perform energy-saving operation on each non-dominated solution in the non-dominated solution set π;

[0017] S2.7. Generate high-level individuals based on the Q table, where each high-level individual is composed of a low-level heuristic sequence;

[0018] S2.8. Apply each high-level individual to π, calculate the contribution rate of each high-level individual, and select the elite high-level individual.

[0019] S2.9, apply the elite high-level individuals to Π, update the Q-table and non-dominated solution set Π;

[0020] S2.10. Determine whether the termination condition (iteration number = 50) is met. If so, output the non-dominated solution set Π; otherwise, jump to S2.7.

[0021] Specifically, the encoding scheme is as follows: for EE-DAHFSP-PM, each feasible solution (π, V) is represented by a two-dimensional sequence π = {π F ,π P ,π J} and a velocity matrix V, the sequence π has three rows, π F represents the factory sequence, which contains the factory allocation information for all products. π P represents the processing order of all products. π J contains the processing order information of the workpiece to which all products belong. The velocity matrix V contains the processing speed information of each workpiece on different machines in the production stage.

[0022] Specifically, the LLHs pool includes 12 low-level heuristics LLH1-LLH 12 , each of which is defined as a state S, and the operation taken by the state is called an action a. In the Q table, the state s and the action a form an element, and the Q table is composed of 12 low-level heuristics.

[0023] Specifically, ε T is used as the greedy probability in the greedy strategy. For a specific state, there is a probability of 1-ε T to select the action with the maximum Q value in the Q table, and ε T is calculated as follows:

[0024] where ε0 is the initial greed rate (ε0 = 0.15), ε f is the final greed rate (ε f = 0.01), T total is the total number of iterations of the algorithm, and T now is the current iteration.

[0025] Specifically, in S2.7, the specific steps of generating a high-level individual according to the Q table include:

[0026] S2.7.1, select a state s t according to the SelectAnAction algorithm, get the action a t and the next state s t+1 , and execute the action a t on the non-dominated solution set Π in the population to obtain a new non-dominated solution set Π', and evaluate the performance of Π';

[0027] The specific method for evaluating new solutions is as follows: During the QLHHEA operation, the action selected by Q-learning is applied to the Pareto solution set. The value of the selected action is determined by the quality of the resulting Pareto frontier (PF). The performance of the action is evaluated using the convergence metric (CV) and the diversity metric (DV), which are defined by the following formulas:

[0028]

[0029] Among them, P t Indicates that by state s t Next, perform action a t The obtained Pareto frontier (PF), P * represents the reference Pareto optimal frontier, d i Indicates P t The Euclidean distance between each pair of adjacent points in , represents the average of these distances, and the N variable represents P t The number of non-dominated solutions in the distance d f and d l Represents P t The Euclidean distance between the extreme value solution and the boundary solution, the smaller the CV t The value indicates that P t Stronger convergence, larger DV t The value indicates that P t The diversity of the solution is stronger, so the following four performance evaluation (PE) indicators are defined:

[0030] PE1:DV t <DV t+1 and CV t >CV t+1 ;PE2:DV t <DV t+1 and CV t ≤CV t+1 ;

[0031] PE3:DV t ≥DV t+1 and CV t >CV t+1 ;PE4:DV t ≥DV t+1 and CV t ≤CV t+1 ;

[0032] S2.7.2, set the reward function, update the Q table according to the reward r obtained, and change the state s t Update to the next state s t+1, reward function r(s t ,a t ) and Q-table updating formula are as follows:

[0033]

[0034] S2.7.3, if all high-level individuals are generated, jump to S2.8, otherwise jump to S2.7;

[0035] Specifically, 12 low-level heuristics LLH1-LLH 12 include: 9 product parts and 3 process parts;

[0036] The 9 product parts include: LLH1: key product and non-key product exchange, LLH2: non-key product after inserting key product, LLH3: key product and non-key product reverse, LLH4: product exchange, LLH5: product insertion 1, LLH6: product insertion 2, LLH7: product reverse 1, LLH8: product reverse 2, LLH9: product reverse 3;

[0037] The 3 process parts include: LLH 10 : workpiece exchange, LLH 11 : workpiece insertion 1, LLH 12 : workpiece insertion 2.

[0038] The above technical solutions are adopted in the embodiments of the application to achieve the following technical effects:

[0039] 1. In terms of energy saving and emission reduction: compared with other inventions, the present application can significantly reduce the energy consumption of the mixed flow shop and reduce carbon emissions, meeting the requirements of green development;

[0040] 2. In terms of economic benefits: the present application takes into account the impact of preventive maintenance activities on machines, avoiding damage to machines under unpredictable circumstances. This can help manufacturers complete production plans on time and reduce additional losses;

[0041] 3. QLHHEA as a knowledge-driven learning mechanism helps the strategy space to select suitable low-level heuristics through continuous trial and error and experience accumulation, guiding the search to a better solution space, which is more advantageous than most random search solution space-based algorithms;

[0042] 4. The 12 low-level heuristics proposed by QLHHEA can search the solution space in all directions, effectively expanding the search range;

[0043] 5.The application provides implementation suggestions for preventive maintenance of scheduling problems of machine maintenance in actual industry, not only accurately determines the time point of preventive maintenance of the machine, but also improves the utilization rate of the machine and reduces the failure probability of the machine. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments and together with the description serve to explain exemplary implementations of the application. The illustrated embodiments are exemplary only and not limiting of the scope of the application. In all the drawings, like reference numerals refer to like parts throughout the several views.

[0045] Figure 1 A schematic flow chart of the distributed assembly hybrid flow shop scheduling method based on hyper-heuristic reinforcement learning in the application;

[0046] Figure 2 A specific process schematic diagram of the 12 low-level heuristics in the application;

[0047] Figure 3 A layout diagram of EE-DAHFSP-PM in the application;

[0048] Figure 4 A performance comparison diagram of QLHHEA under 12 scale problems in the application;

[0049] Figure 5 A curve of QLHHEA in the application in Figure 4 A non-dominated point is taken out from the curve and a Gantt chart is drawn.

[0050] Figure 6 A conversion between states and actions in Q learning in the application;

[0051] Figure 7 A principle diagram of the hyper-heuristic evolutionary algorithm QLHHEA based on Q learning in the application. DETAILED DESCRIPTION

[0052] In order to make the purposes, technical solutions and advantages of the application clearer, the application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the application.

[0053] It should be noted that the terms "first", "second", etc. in the embodiments of the present application are only used for the purpose of description and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined as "first", "second" can be explicitly or implicitly included at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the scope of protection claimed in the present application.

[0054] In the description of the present application, it should be understood that the step number before the step does not indicate the order of executing the steps before and after, but only for the convenience of describing the present application and distinguishing each step, therefore it should not be understood as a limitation of the present application.

[0055] The technical solution proposed in the present application is: the Q-learning based super-heuristic evolutionary algorithm (QLHHEA) uses Q-learning as HLS to manage and manipulate LLH to search for the optimal LLH sequence on the heuristic search space. Then, the optimal LLH sequence found is used to search for the optimal pareto solution set on the solution space. The Q-learning based super-heuristic evolutionary algorithm is a double-layer framework, which consists of a high-level strategy (HLS) and a set of low-level heuristics (LLH). Further, in the technical solution of the present application, 12 low-level heuristics are designed, which are defined as states, and the operations taken by the states are called actions.

[0056] The implementation of the distributed assembly hybrid flow shop scheduling method based on super-heuristic reinforcement learning in the technical solution of the present application mainly includes establishing a multi-objective energy-saving distributed assembly hybrid flow shop with machine maintenance mathematical model; proposing a Q-learning based super-heuristic evolutionary algorithm to solve the problem of multi-objective energy-saving distributed assembly hybrid flow shop with machine maintenance. The specific steps are as shown in Figure 1 , which include:

[0057] S1, model construction: according to the scheduling requirements of multi-objective energy-saving distributed assembly hybrid flow shop with machine maintenance, a multi-objective energy-saving distributed assembly hybrid flow shop planning model, i.e. EE-DAHFSP-PM MILP model, is established.

[0058] Specifically, the optimization objective of the EE-DAHFSP-PM MILP model is to minimize the maximum completion time of the factory, and to minimize the total carbon emission of all factories.

[0059] Further, the above mathematical model is specifically: EE-DAHFSP-PM MILP model, the optimization objective of the model is Objective: min{Cmax TCE(1) subject to 40 constraints shown in equations (2)-(41) below. Where C max is the maximum makespan of the factory, and TCE is the total carbon emission of all factories.

[0060] The detailed description of the specific constraints in the MILP model of EE-DAHFSP-PM is shown in the following Example 2.

[0061] S2, solving the MILP model of EE-DAHFSP-PM:

[0062] The super-heuristic evolutionary algorithm based on Q-learning uses Q-learning as HLS to manage and manipulate LLH to search for the optimal LLH sequence on the heuristic search space, and the optimal LLH sequence found is used to search for the optimal pareto solution set on the solution space to obtain the scheduling scheme.

[0063] Specifically, the specific steps of solving the MILP model of EE-DAHFSP-PM in S2 above include:

[0064] S2.1, designing the encoding and decoding scheme of EE-DAHFSP-PM;

[0065] The encoding scheme is as follows: for EE-DAHFSP-PM, each feasible solution (π, V) is represented by a two-dimensional sequence π = {π F , π P , π J} and a velocity matrix V. The sequence π has three rows, π F represents the factory sequence, which contains the factory allocation information for all products. π P represents the processing order of all products. π J contains the processing order information of the workpieces belonging to all products. The velocity matrix V contains the processing speed information of each workpiece on different machines in the production stage.

[0066] It is worth noting that the workpieces belonging to the same product must be allocated in the same factory and the workpiece sequence cannot be split. In the decoding process, we use the Earliest Completion Time Rule (ECT) to allocate the workpieces to the machine that is expected to complete the workpiece processing task earliest. In the production stage, the workpieces under each factory are allocated to the machines in the processing order. When the workpieces belonging to each product are completed, the product enters the transportation and assembly stages in turn. The decoding process of the workpiece (or product) takes into account the age of the machine. If the age of the machine exceeds the threshold, the PM is executed to reset the age of the machine to 0, and then the processing is performed; otherwise, the processing can be performed directly.

[0067] S2.2, constructing a pool of low-level heuristics (LLHs);

[0068] As a classical reinforcement learning algorithm based on Markov decision process, Q-learning algorithm can train an agent to learn the optimal behavior policy in a specific environment and maximize the cumulative reward. Markov decision process can be represented as (S, A, P, r), where S is the set of all possible states of the environment, A is the set of actions that the agent can take at each state, P describes the probability of transitioning to other states after taking an action in a certain state, and finally r gives the immediate reward obtained by taking a specific action in a specific state. In reinforcement learning, the state is the current environment perceived by the agent, and the action is the behavior that the agent can take in the current state.

[0069] In this application, S and A are used to represent the state set and the action set (the state set consists of the pool of LLHs) respectively, and 12 simple and effective low-level heuristics are designed to form LLHs:

[0070] S = {LLH1, LLH2, LLH3, LLH4, LLH5, LLH6, LLH7, LLH8, LLH9, LLH 10 , LLH 11 , LLH 12}

[0071] The action set is the same as the above state set elements, i.e. A = S. Figure 6 The transition between states and actions is shown, and the arrows represent the transition between states or actions.

[0072] The pool of LLHs can be divided into two parts, product and process, and neighborhood search is performed on the product and job sequence respectively. x pos g is the xth position of product P P on π y , and loc g is the yth position of product P i on .

[0073] The 12 low-level heuristics are as follows, and their specific processes are shown in Figure 2 :

[0074] LLH1: Exchange of key products and non-key products: in the key factory f c , randomly select a key product and a non-key product for exchange.

[0075] LLH2: Insert key product after non-key product: in the key factory f cNext, randomly select a key product and a non-key product. Insert the key product after the non-key product.

[0076] LLH3: Key product and non-key product reverse: At key factory f c Next, randomly select a key product and a non-key product. Reverse the product sequence between the key product and the non-key product.

[0077] LLH4: Product exchange: At π P Randomly select two products P g and P g′ , g≠g', P g and P g′ are located at positions pos x and pos x′ , pos x ≠pos x′ . Exchange P g and P g′ from to

[0078] LLH5: Product front insert 1: At π P Randomly select two products P g and P g′ , g≠g', P g and P g′ are located at positions pos x and pos x′ , pos x < pos x′ . Insert P g′ in front of P g from to

[0079] LLH6: Product front insert 2: At π P Randomly select two products P g and P g′ , g≠g', P g and P g′ are located at positions pos x and pos x′ , pos x < pos x′ . Insert P g′ after P g from to

[0080] LLH7: Product reverse 1: At π P Randomly select a product P g , Pg Located at position pos x On. Reverse with P g Two subsequences as the demarcation point and from arrive

[0081] LLH8: Product Inversion 2: At π P Randomly select two products P g With P g′ , g≠g′, P g and P g′ Located at position pos x and pos x′ Up, pos x <pos x′ . g′ With P g Product sequence between Reverse, from arrive

[0082] LLH9: Product Inversion 3: At π P Randomly select two products P g With P g′ , g≠g′, P g and P g′ Located at position pos x and pos x′ Up, pos x <pos x′ . g′ With P g Other product series and Reverse, from arrive

[0083] LLH 10 :Workpiece exchange: randomly select a product P g , in product P g The sequence of the artifact Randomly select two workpieces J i and J i′ , i≠i′ and i,i′∈Ω g .J i and J i′ Located at location loc y and loc y′ Up, loc y ≠loc y′ . Exchange J i and J i′ from to

[0084] LLH 11 : Randomly select a product P g , whose sequence of jobs g belongs to Randomly select two jobs J i and J i′ , i≠i' and i,i'∈Ω g . J i and J i′ are located at loc y and loc y′ , respectively, loc y <loc y′ . Insert J i′ in front of J i from to

[0085] LLH 12 : Randomly select a product P g , whose sequence of jobs g belongs to Randomly select two jobs J i and J i′ , i≠i' and i,i'∈Ω g . J i and J i′ are located at loc y and loc y′ , respectively, loc y <loc y′ . Insert J i′ behind J i from to

[0086] S2.3, initialize the parameters of the hyper-heuristic evolutionary algorithm of Q-learning; the parameters thereof specifically include population size popsize, learning rate λ, discount rate γ, elite high-level individual ratio initial greed rate e0, final greed rate e f .

[0087] S2.4, after initializing the population, screen the population individuals, retain the population non-dominated solutions, and obtain a non-dominated solution set Π.

[0088] S2.5, use the destroy-reconstruct algorithm to improve the quality of the non-dominated solution set Π: for each non-dominated solution in the non-dominated solution set Π, in π PIn the middle, randomly select d products, and completely destroy-reconstruct the workpieces of each product. Store the d products randomly selected in P In the middle, randomly select d products, and completely destroy-reconstruct the workpieces of each product. Store the d products randomly selected in In the middle, sequentially destroy-reconstruct the workpieces to which the products belong, wherein is the sequence of the workpiece to which the current product belongs. The number of destroyed workpieces is set to be the total workpieces of the current product, i.e., n. The construction stage starts at the end of the destruction stage, and the main task is to reinsert the workpieces into the optimal position to build a new feasible and complete workpiece sequence Solve the non-dominated solution set for the initialized population, and take the union set with Π, and then perform the non-dominated operation again to obtain the non-dominated solution set Π.

[0089] S2.6, perform energy-saving operation on each individual in the non-dominated solution set Π: without affecting the overall completion time, this paper proposes an energy-saving strategy, which mainly reflects in reducing energy consumption by adjusting the processing speed of non-critical path jobs. Based on the nature of DAHFSP problem, this paper considers non-critical path speed reduction processing for workpieces in the production stage. First, determine the critical path under each factory; find non-critical workpieces except those on the critical path; then find the workpiece with the minimum processing time on a specific machine among the non-critical workpieces, and finally reduce the speed of the processing machine of this workpiece.

[0090] S2.7, generate high-level individuals according to the Q table, each high-level individual is composed of low-level heuristic sequences;

[0091] Specifically, the generation of high-level individuals specifically includes:

[0092] S2.7.1, select a state s according to the SelectAnAction algorithm t , get action a t and next state s t+1 (s t+1 ←a t ), and execute action a t on the non-dominated solution set Π in the population to obtain a new non-dominated solution set Π', and evaluate the performance of Π';

[0093] The SelectAnAction algorithm is an action selection algorithm. The detailed process is shown in algorithms 1 and 2.

[0094]

[0095]

[0096] where the specific way of evaluating the new solution is that during the running of QLHHEA, the action selected by Q-learning is applied to the Pareto solution set. The value of the selected action is determined by the quality of the generated Pareto front (PF). The performance of the action is evaluated by a convergence index (CV) and a diversity index (DV), which are defined by the following equations, respectively:

[0097]

[0098] where P t represents the Pareto front (PF) obtained by performing action a t in state s t , P * represents the reference Pareto optimal front. d i represents the Euclidean distance between each pair of adjacent points in P t , and represents the average of these distances. The variable N represents the number of non-dominated solutions in P t . The distances d f and d l represent the Euclidean distances between the extreme solutions and the boundary solutions in P t , respectively. A smaller CV t value indicates stronger convergence of P t , and a larger DV t value indicates stronger diversity of the solutions in P t .

[0099] Therefore, we define the following four performance evaluation (PE) indexes:

[0100] PE1: DV t < DV t+1 and CV t > CV t+1 , PE2: DV t < DV t+1 and CV t ≤ CV t+1 ,

[0101] PE3: DV t ≥ DV t+1 and CV t > CV t+1 , PE4: DV t ≥ DV t+1 and CV t ≤ CV t+1 .

[0102] In this application, we use ε TAs the greedy probability in the improved greedy strategy; for a specific state, there is 1-ε T The probability of selecting the action with the largest Q value in the Q table is T When the value is small, the algorithm will guide the search towards a more favorable direction. T The calculation formula is as follows:

[0103] Where ε0 is the initial greed rate (ε0=0.15), ε f is the final greed rate (ε f =0.01), T total is the total number of iterations of the algorithm, T now The current iteration round.

[0104] S2.7.2, set the reward function, update the Q table according to the reward r obtained, and change the state s t Update to the next state s t+1 . Reward function r(s t ,a t ) and the Q table update formula are as follows:

[0105]

[0106]

[0107] S2.7.3. If all high-level individuals are generated, jump to S2.8, otherwise jump to S2.7.

[0108] S2.8: Apply each high-level individual to each individual in the population, calculate the contribution rate of each high-level individual, and select the elite high-level individuals

[0109] S2.9: Apply the elite high-level individuals to the non-dominated solution set Π and update the Q table and Π.

[0110] S2.10: Determine whether the termination condition (iteration number = 50) is met. If so, output Π; otherwise, jump to S2.7.

[0111] Example 2

[0112] The MILP model for EE-DAHFSP-PM is as follows:

[0113] The main purpose of establishing the MILP model of EE-DAHFSP-PM and establishing the multi-objective function and a series of constraints is to minimize the maximum completion time of the plant and the total carbon emissions of all plants.

[0114] Table 1 contains the parameters of the mathematical model. Figure 3Figure 1 shows the layout of the EE-DAHFSP-PM. H products P = {P1, P2,..., P H} are assembled by N jobs J = {J1, J2,..., J N}. Each product is composed of different jobs, and each job belongs to only one product. The jobs included in each product are assigned to F f facilities for processing. Each facility is a hybrid flow shop, which includes production, transportation and assembly stages.

[0115] In the production stage, each job needs to complete S production processes in a hybrid flow shop, which includes S sub-production stages. In each sub-production stage, there are multiple unrelated parallel machines, which can realize flexible manufacturing of jobs. Once all jobs of a product have been processed, they will be transferred to the transporters and assembly machines in turn, and then go through the transportation stage and the assembly stage, in which each facility only contains one transporter and one assembly machine.

[0116] The normal processing time of jobs (or products) will be affected after the machine wears out. Therefore, the linear deterioration function is used to describe the deterioration process of the machine in the present application. In order to avoid machine failure caused by wear and tear, preventive maintenance activities are carried out on the machine. Each machine has an initial machine age a. When the machine is working, its machine age will increase. However, the machine age is not allowed to reach the upper limit of the cumulative running time T of the machine. Therefore, in order to avoid this situation, PM activities need to be performed on the machine to restore its machine age to the initial state. The time point of machine PM is not a fixed point set in advance, but is determined by the constraint condition whether to perform PM. On each machine, except for the job (or product) located in the first position, which does not need to be judged for preventive maintenance, the other positions need to be judged in advance. If the machine age exceeds the threshold value after processing the current job (or product), PM needs to be performed before processing. In addition, the energy consumption of the machine is also considered in the DAHFSP. Each machine has d discrete and adjustable processing speeds, i.e. V = {V1, V2,..., V d}. Once a specific job on the machine starts processing, the speed of the machine cannot be changed during the processing time. In addition to considering the processing energy consumption of the machine, the standby energy consumption of the machine is also considered. The energy consumption generated during PM is considered in the standby energy consumption.

[0117] General assumptions: 1) All machines and workpieces are available at time zero; 2) All workpieces should be processed continuously, without preemption or interruption; 3) The processing time is deterministic, and the processing speed of a machine remains constant during the processing of a specific workpiece; 4) At any given time, each workpiece can be processed on at most one machine, and each machine is allowed to process at most one workpiece; 5) Once the factory assignment of a product is determined, each product cannot be transferred between factories, and all operations of the workpieces contained in a product should be performed within the same factory; 6) The start-up time of a machine is not considered; 7) The preventive maintenance (PM) time of all machines is the same; 8) The cumulative run time of a machine does not exceed the upper limit of the cumulative run time T.

[0118] The specific constraints of the MILP model of EE-DAHFSP-PM are described as follows:

[0119] Constraint (2) determines the maximum completion time C of a product max , C max must be greater than or equal to the assembly completion time of any one product in the assembly stage.

[0120] Constraints (3) and (4) are the earliest start times of each product in the transportation and assembly stages.

[0121] Constraint (5) is the earliest start time of any operation of any workpiece in the production stage.

[0122] Constraints (6), (7), and (8) specify the earliest start time constraints for adjacent workpieces (or products) on the same machine in the production, transportation, and assembly stages; when considering machine maintenance, the machine maintenance time is divided into the previous workpiece (or product), i.e., the processing completion time of the previous workpiece (or product) is the end time of the machine maintenance.

[0123] Constraint (9) determines the earliest start time of the first operation of a workpiece; where the start time of the first operation of the first workpiece is 0.

[0124] Constraints (10), (11), and (12) ensure that the initial age of each machine in the three stages is 0.

[0125] Constraints (13), (14), and (15) represent the actual processing time of a workpiece (or product).

[0126] Constraints (16), (17), and (18) represent the update formula for the age of a machine without machine maintenance.

[0127] Constraints (19), (20), and (21) ensure that the age of a machine can be restored to 0 after the machine performs a PM activity.

[0128] Constraints (22), (23) and (24) limit that no PM activity is performed before the first processing operation of each machine.

[0129] Constraints (25), (26) and (27) limit that all workpieces of the same product cannot be split and must be assigned to the same factory.

[0130] Constraint (28) limits that each operation of each workpiece can only be processed on a specific machine of the selected factory.

[0131] Constraints (29), (30) and (31) specify that each machine can only process one workpiece or product at a time.

[0132] Constraints (32), (33) and (34) limit that there is no free position before the scheduled operation of any machine.

[0133] Constraint (35) is the formula for calculating the total carbon emissions, considering the total carbon emissions of the three stages.

[0134] Constraints (36), (37) and (38) determine the positions of the machines for PM activities.

[0135] Constraints (39), (40) and (41) determine the completion time of the workpiece (or product) in the three stages.

[0136] Constraints (2) to (41) are respectively shown in the following expressions (2) to (41):

[0137]

[0138]

[0139]

[0140] Finally, to test the effectiveness of QLHHEA in solving EE-DAHFSP-PM, QLHHEA is compared with Jaya, MOEA / D, Non-dominated Sorting Genetic Algorithm II (NSGA-II), Q-learning, and Variable Neighborhood Search driven NSGA-II (QVNS-NSGA-II). All algorithms are implemented in Python. Each algorithm is independently run on a PC with an Intel(R) Core(TM) i5-12400F@2.50 GHz processor and 16 GB RAM under the Windows 11 operating system. The experimental data are randomly generated, i.e., the number of products = {10, 15, 20, 25, 30}, the number of factories = {2, 3}, the number of production sub-phases = {2, 4, 6}, and the number of machines in each production sub-phase = {3, 4, 5}. The processing time of workpieces (or products) in the production, transportation, and assembly stages is distributed in the interval [1, 100]. The machine deterioration factors of the production, transportation, and assembly stages are 0.1, 0.05, and 0.15, respectively, and the machine maintenance time of the three stages is 10; the upper limit of the cumulative running time of the machine is T = 100. Four controllable parameters are used in the present invention: population size popsize, elite high-level individual proportion learning rate λ, and discount rate γ; the best parameter combination is determined according to the results of multiple experiments: popsize = 100, λ = 0.3, and γ = 0.8. Each test instance is represented as “number of factories-number of products-number of production sub-phases-number of machines per stage”. For example, a test instance with 2 factories, 10 products, 2 production sub-phases, and 3 machines per stage is represented as “2-10-2-3”. Figure 4 The performance comparison of QLHHEA under 12 instance-scale problems is given as follows:

[0141] As shown in Figure 4 , QLHHEA shows better overall performance than other algorithms in all 12 instances, and has an advantage in multi-objective optimization problems.

[0142] In the curve of QLHHEA under the “3-10-2-3” scale in Figure 4 , a non-dominated point is taken out and a Gantt chart is drawn as Figure 5 ​As shown in the figure, product 1 consists of workpieces 1, 2, 3, and 4; product 2 consists of workpieces 5, 6, and 7; product 3 consists of workpieces 8, 9, 10, and 11; product 4 consists of workpieces 13, 14, and 15; product 5 consists of workpieces 16, 17, 18, 19, and 20; product 6 consists of workpieces 21 and 22; product 7 consists of workpieces 23, 24, 25, and 26; product 8 consists of workpieces 27, 28, and 29; product 9 consists of workpieces 30, 31, 32, 33, and 34; and product 10 consists of workpieces 35, 36, 37, 38, and 39. Products 2, 8, and 5 are sequentially assigned to factory 1 for processing; products 1, 4, and 7 are sequentially assigned to factory 2 for processing; and products 9, 6, 3, and 10 are sequentially assigned to factory 3 for processing. The workpieces belonging to each product are processed sequentially and inseparably. In each factory, machines 1 through 3 represent production sub-phase 1, and machines 4 through 6 represent production sub-phase 2. After all workpieces for each product complete the production phase, the product enters the transportation and assembly phases. The gray area in the figure represents machine maintenance. When a machine reaches a threshold T after processing a workpiece (or product), maintenance is required before the next processing.

[0143] Table 1

[0144]

[0145]

[0146]

[0147]

[0148] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented using general-purpose computer devices. They can be concentrated on a single computer device or distributed on a network composed of multiple computer devices. Alternatively, they can be implemented using program codes executable by the computer device, so that they can be stored in a storage device and executed by the computer device. In some cases, the steps shown or described can be performed in a different order than herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0149] It should be noted that the above are only preferred embodiments of the present application and do not limit the scope of patent protection of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the scope of patent protection of the present application.

Claims

1. A distributed assembly hybrid flow shop scheduling method based on hyper-heuristic reinforcement learning, characterized by: include: S1. Construction of MILP model of EE-DAHFSP-PM: Based on the scheduling requirements of a multi-objective energy-saving distributed assembly hybrid flow shop with machine maintenance, a planning model of a multi-objective energy-saving distributed assembly hybrid flow shop with machine maintenance, namely the MILP model of EE-DAHFSP-PM, is established; The optimization objectives of the MILP model of EE-DAHFSP-PM are to minimize the maximum completion time of the plant and, to minimize the total carbon emissions of all plants; S2, EE-DAHFSP-PM MILP model solution: The Q-learning-based hyper-heuristic evolutionary algorithm QLHHEA uses Q-learning as HLS to manage and manipulate LLHs to find the optimal LLH sequence in the heuristic search space. The found optimal LLH sequence is applied to the solution space to search for the optimal Pareto solution set and obtain the scheduling solution. In S2, the specific steps for solving the MILP model of EE-DAHFSP-PM include: S2.

1. Design the encoding and decoding scheme of EE-DAHFSP-PM; S2.

2. Construct a low-level heuristic LLHs pool. The LLHs pool is divided into two parts: products and processes. Neighborhood search is performed on product sequences and workpiece sequences respectively. S2.

3. Initialize the parameters of the Q-learning hyper-heuristic evolutionary algorithm, which specifically includes the population size , learning rate , discount rate , elite advanced individual ratio , initial greed rate , Final Greed Rate ; S2.

4. Initialize the population, screen the individuals in the population through the fitness function, and obtain the non-dominated solution set in the population ; S2.

5. Destruction-Reconstruction: Non-dominated Solution Sets in a Population Perform a destruction-reconstruction operation, that is, Random selection products, and completely destroy and reconstruct the artifacts of each product; the population after destruction and reconstruction is compared with Find the union, search for non-dominated solutions again, and obtain a new non-dominated solution set ; S2.6, Energy-saving strategy: non-dominated solution set Perform energy-saving operations; S2.7, according to The table generates high-level individuals, each of which is composed of a low-level heuristic sequence; S2.8, each high-level individual acts on Calculate the contribution rate of each high-level individual and select the elite high-level individuals ; S2.9, the role of elite high-level individuals in On the Q table and non-dominated solution set, update ; S2.10, determine whether the termination condition is met, the number of iterations = 50, if it is met, then output the non-dominated solution set ; Otherwise, jump to S2.7; In S2.7, according to The specific steps for generating high-level individuals in the table include: S2.7.1, according to The algorithm selects a state , get action and the next state , and the non-dominated solution set in the population Execute an action , obtain a new non-dominated solution set , and evaluate performance; The specific way to evaluate the new solution is as follows: during the operation of QLHHEA, the action selected by Q-learning is applied to the Pareto solution set; the value of the selected action is determined by the quality of the generated Pareto frontier (PF); the performance of the action is determined by the convergence indicator and diversity indicators Conduct an assessment, They are defined by the following formulas: ； ； in, Indicates that the state Next action The obtained Pareto frontier (PF) is represents the reference Pareto optimal frontier, express The Euclidean distance between each pair of adjacent points in , represents the average of these distances, Variable representation The number of non-dominated solutions, the distance and Respectively The Euclidean distance between the extreme value solution and the boundary solution is smaller Value indicates Stronger convergence, larger The value indicates The diversity of the solution is stronger, so the following four performance evaluation (PE) indicators are defined: PE1: PE2: PE3: PE4: ; S2.7.

2. Set the reward function according to the reward obtained Update the Q table and set the state Update to next state , the reward function And the Q table update formula is as follows: ； ； S2.7.

3. If all high-level individuals are generated, jump to S2.8, otherwise jump to S2.

7.

2. The scheduling method according to claim 1, characterized in that: The specific encoding scheme is: for EE-DAHFSP-PM, each feasible solution By two-dimensional sequence and the velocity matrix To express, the sequence There are three lines, Represents the factory sequence, which contains the factory allocation information for all products; It indicates the processing order of all products; Contains the processing sequence information of all products' workpieces; speed matrix It contains information about the processing speed of each workpiece on different machines during the production stage.

3. The scheduling method according to claim 1, characterized in that: The LLHs pool includes 12 low-level heuristics LLH1-LLH 12 , each of which is defined as a state S, and the operation taken by the state is called an action , in the Q table state and action The constituent elements are consistent and consist of 12 low-level heuristics.

4. The scheduling method according to claim 1, characterized in that: use As the greedy probability in the greedy strategy, for a specific state, there is The probability of selecting the action with the largest Q value in the Q table is, The calculation formula is as follows: ,in is the initial greed rate , is the final greed rate , is the total number of algorithm iterations, The current iteration round.

5. The scheduling method according to claim 3, characterized in that: 12 low-level heuristics LLH1-LLH 12 It includes: 9 product parts and 3 process parts; The 9 product sections include: LLH1: key products and non-key products are exchanged, LLH2: non-key products are inserted after key products, LLH3: key products and non-key products are reversed, LLH4: product exchange, LLH5: product is inserted in front 1, LLH6: product is inserted in front 2, LLH7: product is reversed 1, LLH8: product is reversed 2, LLH9: product is reversed 3; The 3 process parts include: LLH 10 : Workpiece exchange, LLH 11 :Workpiece forward insertion 1, LLH 12 : Workpiece inserted forward 2.

Images