Distributed assembly hybrid flow shop scheduling method based on hyper-heuristic reinforcement learning

Abstract

The invention discloses a distributed assembly hybrid flow shop scheduling method based on hyper-heuristic reinforcement learning, and belongs to the technical field of hybrid flow shop scheduling. The method comprises the following steps: constructing an MILP model of EE-DAHFSP-PM: according to scheduling requirements of the multi-target energy-saving distributed assembly hybrid flow shop with machine maintenance, establishing a planning model of the multi-target energy-saving distributed assembly hybrid flow shop with machine maintenance, namely the MILP model of EE-DAHFSP-PM; the optimization objective of the MILP model of the EE-DAHFSP-PM is to minimize the maximum completion time of a factory and minimize the total carbon emission of all factories; and MILP model solving of EE-DAHFSP-PM: based on a Q learning hyper-heuristic evolutionary algorithm QLHHEA, adopting Q learning as an HLS to manage and control LLH to search for an optimal LLH sequence in a heuristic search space, and enabling the searched optimal LLH sequence to act on a solution space to search for an optimal pareto solution set to obtain a scheduling scheme.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of hybrid assembly line workshop scheduling, and in particular to a distributed assembly hybrid assembly line workshop scheduling method based on hyper-heuristic reinforcement learning. Background Art

[0002] In the context of economic globalization and sustainable industrial development, green industry has become a popular trend of new industrialization. As a modern energy-saving and environmentally friendly manufacturing model, low-carbon manufacturing can promote the upgrading and transformation of green industry. Energy consumption leads to the release of a large amount of greenhouse gases. This situation has aggravated climate change and made energy conservation and emission reduction more important. In addition, with the rapid development of intelligent manufacturing and low-carbon manufacturing, distributed manufacturing can respond to rapidly changing markets, improve resource utilization and production efficiency, and make positive contributions to sustainable green industry. At the same time, many scholars have begun to conduct extensive and in-depth research on the distributed assembly hybrid flow shop scheduling problem (DAHFSP). DAHFSP has been proven to be a strong NP-hard problem, and it is too challenging to implement a robust and high-quality scheduling solution to solve this problem in a short time. Hyperheuristic algorithm (HHA) is a class of advanced automation algorithms proposed in recent years to solve complex computational search problems. At present, researchers have widely used hyperheuristic methods to solve various workshop scheduling problems, such as hybrid workshop scheduling problem, dynamic job shop scheduling problem, flexible job shop scheduling problem, distributed assembly flow shop scheduling problem, hybrid flow shop scheduling problem, etc. This makes HHA a promising algorithmic framework for solving shop scheduling problems.

[0003] The study found that there are some problems with the existing technology: First, most DAHFSPs ignore the energy consumption problem. Low-carbon manufacturing is a common goal pursued by modern enterprises, which requires the industry to pay attention to the sustainability of development and energy conservation and emission reduction. Therefore, in order to ensure product quality and production efficiency, in addition to considering economic benefit indicators, energy consumption indicators need to be further considered. Secondly, most studies are almost based on the idea that machines are always available and have a long life cycle, without considering the actual production situation. In fact, machines will wear and damage during normal use, which causes the machines to be unavailable for a certain period of time. This situation will disrupt production plans, causing delivery delays, additional costs, reduced customer satisfaction and other adverse effects. Therefore, it is of great practical significance and potential value to study the energy-saving distributed assembly hybrid flow shop problem with machine maintenance in an actual production environment. For the energy-saving distributed assembly hybrid flow shop problem with machine maintenance (EE-DAHFSP-PM), the hyper-heuristic evolutionary algorithm based on Q learning (QLHHEA) can provide appropriate strategies to select appropriate search behaviors and guide search trends to promising areas to improve solution efficiency and quality and reduce energy consumption. Summary of the invention

[0004] In order to overcome the technical defects in the above-mentioned existing scheduling schemes, an embodiment of the present application provides a distributed assembly hybrid flow shop scheduling method based on hyper-heuristic reinforcement learning, which determines the factory allocation of products in EE-DAHFSP-PM, the processing order of products in the factory, the processing order of workpieces to which the products belong, and the maintenance time point of each machine, so as to minimize the maximum completion time of all factories and minimize the carbon emissions of all factories.

[0005] The technical solution of the scheduling method is as follows:

[0006] S1. Construction of MILP model of EE-DAHFSP-PM: According to the scheduling requirements of multi-objective energy-saving distributed assembly hybrid flow shop with machine maintenance, a planning model of multi-objective energy-saving distributed assembly hybrid flow shop with machine maintenance, namely, the MILP model of EE-DAHFSP-PM, is established;

[0007] The optimization objectives of the MILP model of EE-DAHFSP-PM are to minimize the maximum completion time of the plant and, to minimize the total carbon emissions of all plants;

[0008] S2, EE-DAHFSP-PM MILP model solution:

[0009] The Q-learning-based hyper-heuristic evolutionary algorithm QLHHEA uses Q-learning as HLS to manage and manipulate LLH to find the optimal LLH sequence in the heuristic search space, and then applies the found optimal LLH sequence to the solution space to search for the optimal Pareto solution set and obtain a scheduling plan.

[0010] Specifically, in S2, the specific steps of solving the MILP model of EE-DAHFSP-PM include:

[0011] S2.1, design the encoding and decoding scheme of EE-DAHFSP-PM;

[0012] S2.2, build a low-level heuristic LLHs pool. The LLHs pool can be divided into two parts: products and processes, and perform neighborhood searches on product sequences and workpiece sequences respectively;

[0013] S2.3. Initialize the parameters of the hyper-heuristic evolutionary algorithm for Q learning, including population size popsize, learning rate λ, discount rate γ, elite high-level individual ratio Initial greed rate e0, final greed rate e f ;

[0014] S2.4, after initializing the population, screen the individuals in the population through the fitness function to obtain the non-dominated solution set Π in the population;

[0015] S2.5, Destruction-Reconstruction: Perform a destruction-reconstruction operation on the non-dominated solution set π. P Randomly select d products from the set, and completely destroy and reconstruct the workpiece of each product; find the union of the destroyed and reconstructed population and Π, and search for non-dominated solutions again to obtain a new non-dominated solution set Π;

[0016] S2.6, energy-saving operation: perform energy-saving operation on each non-dominated solution in the non-dominated solution set Π;

[0017] S2.7, generate high-level individuals according to the Q table, each high-level individual is composed of a low-level heuristic sequence;

[0018] S2.8. Apply each high-level individual to Π, calculate the contribution rate of each high-level individual, and select the elite high-level individuals.

[0019] S2.9, apply the elite high-level individuals to Π, update the Q-table and non-dominated solution set Π;

[0020] S2.10. Determine whether the termination condition (iteration number = 50) is met. If so, output the non-dominated solution set Π; otherwise, jump to S2.7.

[0021] Specifically, the encoding scheme is as follows: for EE-DAHFSP-PM, each feasible solution (π, V) is represented by a two-dimensional sequence π = {π F ,π P ,π J} and the velocity matrix V, the sequence π has three rows, π F Represents the factory sequence, which contains the factory allocation information for all products. P It indicates the processing order of all products. J It contains the processing sequence information of all the workpieces of the product. The speed matrix V contains the processing speed information of each workpiece on different machines during the production stage.

[0022] Specifically, the LLHs pool includes 12 low-level heuristics LLH1-LLH 12 , each of which is defined as a state S, and the operation taken by the state is called action a. In the Q table, the state s is consistent with the elements of action a, which consists of 12 low-level heuristics.

[0023] Specific optional, using ε T As the greedy probability in the greedy strategy, for a specific state, there is 1-ε T The probability of selecting the action with the largest Q value in the Q table, ε T The calculation formula is as follows:

[0024] Where ε0 is the initial greed rate (ε0 = 0.15), ε f is the final greed rate (ε f =0.01), T total is the total number of iterations of the algorithm, T now The current iteration round.

[0025] Specifically, in S2.7, the specific steps of generating high-level individuals according to the Q table include:

[0026] S2.7.1. Select a state s according to the SelectAnAction algorithm t , get action a t and the next state s t+1 , and perform action a on the non-dominated solution set Π in the population t , obtain a new non-dominated solution set Π′ and evaluate the performance of Π′;

[0027] The specific way to evaluate the new solution is as follows: During the operation of QLHHEA, the action selected by Q-learning is applied to the Pareto solution set. The value of the selected action is determined by the quality of the generated Pareto frontier (PF). The performance of the action is evaluated by the convergence index (CV) and the diversity index (DV), and CV and DV are defined by the following formulas respectively:

[0028]

[0029] Among them, P t Indicates that by in state s t Next, perform action a t The obtained Pareto frontier (PF), P * represents the reference Pareto optimal frontier, d i Indicates P t The Euclidean distance between each pair of adjacent points in , represents the average of these distances, and the N variable represents P t The number of non-dominated solutions in the distance d f and d l Respectively represent P t The Euclidean distance between the extreme value solution and the boundary solution, the smaller the CV t The value indicates that P t Stronger convergence, larger DV t The value indicates that P t The diversity of the solution is stronger, so the following four performance evaluation (PE) indicators are defined:

[0030] PE1:DV t <DV t+1 and CV t >CV t+1 ;PE2:DV t <DV t+1 and CV t ≤CV t+1 ;

[0031] PE3:DV t ≥DV t+1 and CV t >CV t+1 ;PE4:DV t ≥DV t+1 and CV t ≤CV t+1 ;

[0032] S2.7.2, set the reward function, update the Q table according to the reward r obtained, and change the state s t Update to the next state s t+1, reward function r(s t ,a t ) and the Q table update formula are as follows:

[0033]

[0034] S2.7.3, if all high-level individuals are generated, jump to S2.8, otherwise jump to S2.7;

[0035] Specific optional, 12 low-level heuristics LLH1-LLH 12 It includes: 9 product parts and 3 process parts;

[0036] The 9 product sections include: LLH1: key products and non-key products are exchanged, LLH2: non-key products are inserted after key products, LLH3: key products and non-key products are reversed, LLH4: product exchange, LLH5: product front insertion 1, LLH6: product front insertion 2, LLH7: product reverse 1, LLH8: product reverse 2, LLH9: product reverse 3;

[0037] The 3 process parts include: LLH 10 : Workpiece exchange, LLH 11 : Workpiece forward insertion 1, LLH 12 : Workpiece forward insertion 2.

[0038] The above technical solution can achieve the following technical effects in the embodiments of the present application:

[0039] 1. In terms of energy conservation and emission reduction: Compared with other inventions, the present invention can significantly reduce the energy consumption of the mixed flow workshop and reduce carbon emissions, which meets the requirements of current green development;

[0040] 2. In terms of economic benefits: The present invention takes into account the impact of preventive maintenance activities on the machine, avoiding damage to the machine in unexpected circumstances. This can help manufacturers complete production plans on time and reduce additional losses;

[0041] 3. As a knowledge-driven learning mechanism, QLHHEA helps the policy space select appropriate low-level heuristics through continuous trial and error and accumulated experience, guiding the search towards a better solution space, which is more advantageous than most algorithms based on random search solution space;

[0042] 4. The 12 low-level and medium-level heuristics proposed by QLHHEA can perform perturbation search on the solution space in an all-round way, effectively expanding the search scope;

[0043] 5. The present invention provides implementation suggestions for preventive maintenance for scheduling problems that require machine maintenance in actual industry. It not only accurately determines the time point for preventive maintenance of the machine, but also improves the utilization rate of the machine and reduces the probability of machine failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0045] Figure 1 A schematic flowchart of a distributed assembly hybrid flow shop scheduling method based on hyper-heuristic reinforcement learning in this application;

[0046] Figure 2 Schematic diagram of the specific process of 12 low-level heuristics in this application;

[0047] Figure 3 This is the layout diagram of EE-DAHFSP-PM in this application;

[0048] Figure 4 This is a performance comparison chart of QLHHEA under 12 scale problems in this application;

[0049] Figure 5 For this application Figure 4 Take a non-dominated point from the QLHHEA curve and draw a Gantt chart.

[0050] Figure 6 It is the transition between states and actions in Q learning in this application;

[0051] Figure 7 This is a schematic diagram of the Q-learning-based hyper-heuristic evolutionary algorithm QLHHEA in this application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0053] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0054] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0055] The technical solution proposed in this application is: the Q-learning-based hyper-heuristic evolutionary algorithm (QLHHEA) uses Q-learning as HLS to manage and manipulate LLH to find the optimal LLH sequence in the heuristic search space. Then, the optimal LLH sequence found is applied to the solution space to search for the optimal pareto solution set. The Q-learning-based hyper-heuristic evolutionary algorithm is a two-layer framework consisting of a high-level strategy (HLS) and a set of low-level heuristics (LLH). Furthermore, 12 low-level heuristics are designed in the technical solution of this application, which are defined as states, and the operations taken by the states are called actions.

[0056] The implementation of the distributed assembly hybrid flow shop scheduling method based on hyper-heuristic reinforcement learning in the technical solution of this application mainly includes establishing a mathematical model of a multi-objective energy-saving distributed assembly hybrid flow shop with machine maintenance; proposing a hyper-heuristic evolutionary algorithm based on Q learning to solve the problem of a multi-objective energy-saving distributed assembly hybrid flow shop with machine maintenance. The specific steps are as follows: Figure 1 As shown, including:

[0057] S1. Model construction: According to the scheduling requirements of a multi-objective energy-saving distributed assembly hybrid flow shop with machine maintenance, a planning model of a multi-objective energy-saving distributed assembly hybrid flow shop with machine maintenance, namely, the MILP model of EE-DAHFSP-PM, is established.

[0058] Specifically, the optimization objectives of the MILP model of the EE-DAHFSP-PM are to minimize the maximum completion time of the plant and to minimize the total carbon emissions of all plants.

[0059] Furthermore, the mathematical model is specifically: EE-DAHFSP-PM MILP model, the optimization objective of the model is: min{Cmax , TCE(1), its constraints are shown in the following equations (2)-(41), a total of 40. max is the maximum completion time of the factory, and TCE is the total carbon emissions of all factories.

[0060] For a detailed description of the specific constraints in the MILP model of EE-DAHFSP-PM, please refer to the second embodiment below.

[0061] S2, EE-DAHFSP-PM MILP model solution:

[0062] A hyper-heuristic evolutionary algorithm based on Q-learning adopts Q-learning as HLS to manage and manipulate LLH to find the optimal LLH sequence in the heuristic search space, and then applies the found optimal LLH sequence to the solution space to search for the optimal Pareto solution set and obtain the scheduling plan.

[0063] Specifically, the specific steps for solving the MILP model of EE-DAHFSP-PM in S2 above include:

[0064] S2.1, design the encoding and decoding scheme of EE-DAHFSP-PM;

[0065] The specific encoding scheme is as follows: For EE-DAHFSP-PM, each feasible solution (π, V) is represented by a two-dimensional sequence π = {π F ,π P ,π J} and the velocity matrix V, the sequence π has three rows, π F Represents the factory sequence, which contains the factory allocation information for all products. P It indicates the processing order of all products. J It contains the processing sequence information of all the workpieces of the product. The speed matrix V contains the processing speed information of each workpiece on different machines during the production stage.

[0066] It is worth noting that the workpieces of the same product must be assigned to the same factory and the workpiece sequence cannot be split. In the decoding process, we use the Earliest Completion Time Rule (ECT) to assign the workpiece to the machine that is expected to complete the workpiece processing task the earliest. In the production stage, the workpieces under each factory are assigned to the machines in order of processing. When the workpieces of each product are completed, the product enters the transportation and assembly stages in turn. The decoding process of the workpiece (or product) must take into account the age of the machine. If the age of the machine exceeds the threshold, PM must be executed to reset the age of the machine to 0 before processing; otherwise, processing can be carried out directly.

[0067] S2.2, construct a low-level heuristic LLHs pool;

[0068] As a classic reinforcement learning algorithm based on Markov decision process, Q-learning algorithm can train intelligent agents to learn optimal behavior strategies in a specific environment and maximize cumulative rewards. Markov decision process can be expressed as (S, A, P, r), where S is the set of all possible states of the environment, A is the set of actions that the intelligent agent can take in each state, P describes the probability of transferring to other states after taking a certain action in a certain state, and finally r gives the immediate reward for taking a specific action in a specific state. In reinforcement learning, the state is the current environment perceived by the intelligent agent, and the action is the behavior that the intelligent agent can take in the current state.

[0069] In this application, S and A are used to represent the state set and action set respectively (the state set consists of a pool of LLHs), and 12 simple and effective low-level heuristics are designed to form LLHs:

[0070] S={LLH1, LLH2, LLH3, LLH4, LLH5, LLH6, LLH7, LLH8, LLH9, LLH 10 ,LLH 11 ,LLH 12}

[0071] The action set has the same elements as the state set above, ie A=S. Figure 6 It shows the transitions between states and actions, and arrows indicate transfers between states or actions.

[0072] The LLHs pool can be divided into two parts: products and processes, and the neighborhood search is performed on product and workpiece sequences respectively. x For product P g In π P The xth position on the y For product P g Workpiece J i exist The y-th position on .

[0073] The 12 low-level heuristics are as follows. The specific process is as follows Figure 2 As shown:

[0074] LLH1: Exchange of critical products with non-critical products: in critical factories c Next, a key product and a non-key product are randomly selected for exchange.

[0075] LLH2: Non-critical products are inserted into key products later: in key factories cNext, randomly select a key product and a non-key product. Insert the key product after the non-key product.

[0076] LLH3: Reversal of key products and non-key products: In key factories c In this case, randomly select a key product and a non-key product. Reverse the product sequence between the key product and the non-key product.

[0077] LLH4: Product Exchange: In π P Randomly select two products P g With P g′ , g≠g′, P g and P g′ Located at position pos x and pos x′ Up, pos x ≠pos x′ . Exchange P g and P g′ from arrive

[0078] LLH5: Product Pre-insertion 1: In π P Randomly select two products P g With P g′ , g≠g′, P g and P g′ Located at position pos x and pos x′ Up, pos x <pos x′ . g′ Insert into P g In front, from arrive

[0079] LLH6: Product Pre-insertion 2: In π P Randomly select two products P g With P g′ , g≠g′, P g and P g′ Located at position pos x and pos x′ Up, pos x <pos x′ . g′ Insert into P g Behind, from arrive

[0080] LLH7: Product Inversion 1: At π P Randomly select a product P g , Pg Located at position pos x On. Reverse with P g Two subsequences of the demarcation point and from arrive

[0081] LLH8: Product Inversion 2: At π P Randomly select two products P g With P g′ , g≠g′, P g and P g′ Located at position pos x and pos x′ Up, pos x <pos x′ . g′ With P g Product sequence between Invert, from arrive

[0082] LLH9: Product Inversion 3: At π P Randomly select two products P g With P g′ , g≠g′, P g and P g′ Located at position pos x and pos x′ Up, pos x <pos x′ . g′ With P g Other product series and Invert, from arrive

[0083] LLH 10 :Workpiece exchange: Randomly select a product P g , in product P g The sequence of the artifact Randomly select two workpieces J i and J i′ , i≠i′ and i,i′∈Ω g .J i and J i′ Located at position loc y and loc y′ Up, loc y ≠loc y′ . Exchange J i and J i′ from arrive

[0084] LLH 11 :Workpiece forward insertion 1: Randomly select a product P g , in product P g The sequence of the artifact Randomly select two workpieces J i and J i′ , i≠i′ and i,i′∈Ω g .J i and J i′ Located at position loc y and loc y′ Up, loc y <loc y′ . i′ Insert to J i In front, from arrive

[0085] LLH 12 :Workpiece forward insertion 2: Randomly select a product P g , in product P g The sequence of the artifact Randomly select two workpieces J i and J i′ , i≠i′ and i,i′∈Ω g .J i and J i′ Located at position loc y and loc y′ Up, loc y <loc y′ . i′ Insert to J i Behind, from arrive

[0086] S2.3. Initialize the parameters of the hyper-heuristic evolutionary algorithm for Q learning; its parameters specifically include population size popsize, learning rate λ, discount rate γ, elite high-level individual ratio Initial greed rate e0, final greed rate e f .

[0087] S2.4. After initializing the population, screen the individuals in the population, retain the non-dominated solutions of the population, and obtain the non-dominated solution set Π.

[0088] S2.5. Use the destruction-reconstruction algorithm to improve the quality of the non-dominated solution set Π: For each non-dominated solution in the non-dominated solution set Π, PRandomly select d products from the set and perform complete destruction-reconstruction on the artifacts of each product. P The d products randomly selected from In the process, the artifacts of the products are destroyed and reconstructed one by one. is the sequence of artifacts belonging to the current product. The number of destroyed artifacts is set to all artifacts of the current product, that is, n. The construction phase begins at the end of the destruction phase. The main task is to reinsert the artifacts into the best position and build a new feasible and complete artifact sequence. Solve the non-dominated solution set for the initialized population and find the union with Π, then perform the non-dominated operation again to obtain the non-dominated solution set Π.

[0089] S2.6. Perform energy-saving operations on each individual in the non-dominated solution set Π: Without affecting the overall completion time, this paper proposes an energy-saving strategy, which is mainly reflected in reducing energy consumption by adjusting the processing speed of jobs on non-critical paths. Based on the properties of the DAHFSP problem, this paper considers slowing down the non-critical paths of workpieces in the production stage. First, determine the critical path under each factory; find the non-critical workpieces except the workpieces on the critical path; then find the workpiece with the shortest processing time on a specific machine among the non-critical workpieces, and finally slow down the processing machine of this workpiece.

[0090] S2.7, generate high-level individuals according to the Q table, each high-level individual is composed of a low-level heuristic sequence;

[0091] Specifically, the generation of high-level individuals includes:

[0092] S2.7.1. Select a state s according to the SelectAnAction algorithm t , get action a t and the next state s t+1 (s t+1 ←a t ), and perform action a on the non-dominated solution set Π in the population t , obtain a new non-dominated solution set Π′ and evaluate the performance of Π′;

[0093] The SelectAnAction algorithm is an action selection algorithm. The detailed process is shown in Algorithms 1 and 2.

[0094]

[0095]

[0096] The specific way to evaluate the new solution is as follows: During the operation of QLHHEA, the action selected by Q-learning is applied to the Pareto solution set. The value of the selected action is determined by the quality of the generated Pareto frontier (PF). The performance of the action is evaluated by the convergence index (CV) and the diversity index (DV), and CV and DV are defined by the following formulas respectively:

[0097]

[0098] Among them, P t Indicates that by in state s t Next, perform action a t The obtained Pareto frontier (PF), P * Denotes the reference Pareto optimal frontier. i Indicates P t The Euclidean distance between each pair of adjacent points in , represents the average of these distances. The N variable represents P t The number of non-dominated solutions in the distance d f and d l Respectively represent P t The Euclidean distance between the extreme value solution and the boundary solution. Smaller CV t The value indicates that P t Stronger convergence, larger DV t The value indicates that P t The diversity of Chinese solutions is stronger.

[0099] Therefore, we define the following four performance evaluation (PE) metrics:

[0100] PE1:DV t <DV t+1 and CV t >CV t+1 , PE2:DV t <DV t+1 and CV t ≤CV t+1 ,

[0101] PE3:DV t ≥DV t+1 and CV t >CV t+1 , PE4:DV t ≥DV t+1 and CV t ≤CV t+1 .

[0102] In this application, ε TAs the greedy probability in the improved greedy strategy; for a specific state, there is 1-ε T The probability of selecting the action with the maximum Q value in the Q table is T When the value is small, the algorithm will guide the search in a more favorable direction. T The calculation formula is as follows:

[0103] Where ε0 is the initial greed rate (ε0 = 0.15), ε f is the final greed rate (ε f =0.01), T total is the total number of iterations of the algorithm, T now The current iteration round.

[0104] S2.7.2, set the reward function, update the Q table according to the reward r obtained, and change the state s t Update to the next state s t+1 . Reward function r(s t ,a t ) and the Q table update formula are as follows:

[0105]

[0106]

[0107] S2.7.3. If all high-level individuals are generated, jump to S2.8, otherwise jump to S2.7.

[0108] S2.8: Apply each high-level individual to each individual in the population, calculate the contribution rate of each high-level individual, and select the elite high-level individuals

[0109] S2.9: Apply the elite high-level individuals to the non-dominated solution set Π and update the Q table and Π.

[0110] S2.10: Determine whether the termination condition (iteration number = 50) is met. If so, output Π; otherwise, jump to S2.7.

[0111] Embodiment 2

[0112] The MILP model for EE-DAHFSP-PM is as follows:

[0113] The main purpose of establishing the MILP model of EE-DAHFSP-PM and establishing the multi-objective function and a series of constraints is to minimize the maximum completion time of the plant and the total carbon emissions of all plants.

[0114] Table 1 contains the parameters of the mathematical model. Figure 3is the EE-DAHFSP-PM layout diagram. H products P = {P1, P2, ..., P H} consists of N workpieces J = {J1, J2, ..., J N Each product is composed of different artifacts, and each artifact belongs to only one product. The artifacts included in each product are assigned to F f Processing is carried out in the same factory. Each factory is a mixed flow workshop, including three stages: production, transportation and assembly.

[0115] In the production stage, each workpiece needs to complete S production processes in a mixed flow shop containing S sub-production stages. In each sub-production stage, there are multiple unrelated parallel machines, which can realize flexible manufacturing of workpieces. Once all the workpieces of the product have been processed, they will be transferred to the conveyor and assembly machine in turn, and then go through the transportation stage and assembly stage, where each factory contains only one conveyor and one assembly machine.

[0116] After the machine is worn, the normal processing time of the workpiece (or product) will be affected. Therefore, the linear degradation function is used in the present invention to describe the degradation process of the machine. In addition, in order to avoid machine failures caused by wear, preventive maintenance activities are performed on the machine. Each machine is set to have an initial machine age a. When the machine is working, its machine age will increase. However, the machine age is not allowed to reach the upper limit of the machine's cumulative operating time T. Therefore, in order to avoid this situation, it is necessary to perform PM activities on the machine to restore its machine age to the initial state. The time point when the machine executes PM is not a pre-set fixed point, but whether to execute PM is determined by constraints. On each machine, except for the workpiece (or product) at the first position, which does not require preventive maintenance judgment, other positions need to be judged in advance. If the machine age exceeds the threshold after the current workpiece (or product) is processed, PM needs to be performed before this processing. In addition, DAHFSP also takes into account the energy consumption of the machine. Each machine has d discrete and adjustable processing speeds, that is, V = {V1, V2, ..., V d Once a specific workpiece on a machine begins processing, the speed of the machine cannot be changed during this processing time. In addition to considering the processing energy consumption of the machine, the standby energy consumption of the machine is also considered. The energy consumption generated during PM is taken into account in the standby energy consumption.

[0117] General assumptions: 1) All machines and workpieces are available at time zero; 2) All workpieces should be processed continuously without preemption or interruption; 3) The processing time is deterministic, and the processing speed of the machine remains unchanged during the processing of a specific workpiece; 4) At any given time, each workpiece can be processed on at most one machine, and each machine is allowed to process no more than one workpiece; 5) Once the factory allocation of the product is determined, each product cannot be transferred between factories, and all operations on the workpieces contained in each product should be performed in the same factory; 6) The startup time of the machine is not considered; 7) The preventive maintenance (PM) time of all machines is the same; 8) The cumulative running time of the machine does not exceed the cumulative running time upper limit T.

[0118] The specific constraints of the MILP model of EE-DAHFSP-PM are described as follows:

[0119] Constraint (2) determines the maximum completion time C of the product max , C max It must be greater than or equal to the assembly completion time of any product in the assembly stage.

[0120] Constraints (3) and (4) are the earliest start time for each product in the transportation and assembly stages.

[0121] Constraint (5) is the earliest start time of any operation to which any workpiece belongs in the production stage.

[0122] Constraints (6), (7), and (8) specify the earliest start time constraints for adjacent workpieces (or products) on the same machine during the production, transportation, and assembly stages. When considering machine maintenance, the machine maintenance time is allocated to the previous workpiece (or product), that is, the processing completion time of the previous workpiece (or product) is the end time of machine maintenance.

[0123] Constraint (9) determines the earliest start time of the first operation of a workpiece; where the start time of the first operation of the first workpiece is 0.

[0124] Constraints (10), (11) and (12) ensure that the initial age of each machine in the three stages is 0.

[0125] Constraints (13), (14) and (15) represent the actual processing time of the workpiece (or product).

[0126] Constraints (16), (17) and (18) represent the update formula of machine age in the absence of machine maintenance.

[0127] Constraints (19), (20) and (21) ensure that the age of a machine can be restored to 0 after the machine implements the PM activity.

[0128] Constraints (22), (23), and (24) restrict each machine from performing PM activities before its first processing operation.

[0129] Constraints (25), (26) and (27) restrict all workpieces of the same product to be inseparable and must be assigned to the same factory.

[0130] Constraint (28) restricts each operation of each workpiece to be processed only on a specific machine in the selected factory.

[0131] Constraints (29), (30), and (31) stipulate that each machine can only process one workpiece or product at a time.

[0132] Constraints (32), (33) and (34) restrict any machine from having any vacant position immediately preceding the position where the machine is scheduled to operate.

[0133] Constraint (35) is the total carbon emissions calculation formula, which considers the total carbon emissions of the three stages.

[0134] Constraints (36), (37), and (38) determine the locations where the machine performs PM activities.

[0135] Constraints (39), (40) and (41) determine the completion time of the workpiece (or product) in the three stages.

[0136] Constraints (2) to (41) are respectively expressed as the following expressions (2) to (41):

[0137]

[0138]

[0139]

[0140] Finally, to test the effectiveness of the QLHHEA algorithm in solving the EE-DAHFSP-PM problem; QLHHEA was compared with Jaya, MOEA / D, non-dominated sorting genetic algorithm NSGA-II, Q-learning and variable neighborhood search driven NSGA-II (QVNS-NSGA-II). All algorithms were implemented in Python. Each algorithm was independently run on a PC with an Intel(R) Core(TM) i5-12400F@2.50GHz processor and 16GB RAM under Windows 11 operating system. The experimental data were randomly generated, i.e., the number of products = {10,15,20,25,30}, the number of factories = {2,3}, the number of production sub-stages = {2,4,6}, and the number of machines in each production sub-stage = {3,4,5}. The processing time of the workpieces (or products) in the production, transportation and assembly stages is distributed in the interval [1,100]. The machine loss factors in the production stage, transportation stage, and assembly stage are 0.1, 0.05, and 0.15 respectively, and the machine maintenance time in the three stages is 10; the upper limit of the cumulative running time of the machine is T = 100. The present invention uses four controllable parameters: population size popsize, elite high-level individual ratio Learning rate λ, discount rate γ; determine the best parameter combination based on multiple sets of experimental results: popsize = 100, λ = 0.3, γ = 0.8. Each test case is represented as “number of factories - number of products - number of production sub-stages - number of machines in each stage”. For example, a test case with 2 factories, 10 products, 2 production sub-stages and 3 machines in each stage is represented as “2-10-2-3”. Figure 4 The performance comparison of QLHHEA under 12 instance scale problems is given:

[0141] like Figure 4 As shown in the figure, in all 12 examples, the overall performance of the QLHHEA algorithm is better than other algorithms and it has an advantage in multi-objective optimization problems.

[0142] exist Figure 4 Take a non-dominated point from the curve of QLHHEA under the "3-10-2-3" scale and draw a Gantt chart as follows Figure 5As shown. Product 1 is composed of workpieces 1, 2, 3, and 4; product 2 is composed of workpieces 5, 6, and 7; product 3 is composed of workpieces 8, 9, 10, and 11; product 4 is composed of workpieces 13, 14, and 15; product 5 is composed of workpieces 16, 17, 18, 19, and 20; product 6 is composed of workpieces 21 and 22; product 7 is composed of workpieces 23, 24, 25, and 26; product 8 is composed of workpieces 27, 28, and 29; product 9 is composed of workpieces 30, 31, 32, 33, and 34; product 10 is composed of workpieces 35, 36, 37, 38, and 39. Products 2, 8, and 5 are sequentially assigned to factory 1 for processing; products 1, 4, and 7 are sequentially assigned to factory 2 for processing; products 9, 6, 3, and 10 are sequentially assigned to factory 3 for processing. The workpieces of each product are processed sequentially in the processing order and are inseparable. In each factory, machines 1 to 3 are production sub-stage 1, and machines 4 to 6 are production sub-stage 2. After all the workpieces of each product have completed the production stage, the products enter the transportation and assembly stages in turn. The gray part in the figure represents machine maintenance. When the machine has processed the current workpiece (or product), its machine age reaches the threshold T, and the machine needs to be maintained before the current processing.

[0143] Table 1

[0144]

[0145]

[0146]

[0147]

[0148] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0149] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A distributed assembly hybrid flow shop scheduling method based on hyper-heuristic reinforcement learning, characterized in that: include: S1. Construction of MILP model of EE-DAHFSP-PM: According to the scheduling requirements of multi-objective energy-saving distributed assembly hybrid flow shop with machine maintenance, a planning model of multi-objective energy-saving distributed assembly hybrid flow shop with machine maintenance, namely, the MILP model of EE-DAHFSP-PM, is established; The optimization objectives of the MILP model of EE-DAHFSP-PM are to minimize the maximum completion time of the plant and, to minimize the total carbon emissions of all plants; S2, EE-DAHFSP-PM MILP model solution: The Q-learning-based hyper-heuristic evolutionary algorithm QLHHEA uses Q-learning as HLS to manage and manipulate LLH to find the optimal LLH sequence in the heuristic search space, and then applies the found optimal LLH sequence to the solution space to search for the optimal Pareto solution set and obtain a scheduling plan.

2. The scheduling method according to claim 1, characterized in that: In S2, the specific steps of solving the MILP model of EE-DAHFSP-PM include: S2.1, design the encoding and decoding scheme of EE-DAHFSP-PM; S2.2, build a low-level heuristic LLHs pool. The LLHs pool can be divided into two parts: products and processes, and perform neighborhood searches on product sequences and workpiece sequences respectively; S2.

3. Initialize the parameters of the hyper-heuristic evolutionary algorithm for Q learning, including population size popsize, learning rate λ, discount rate γ, elite high-level individual ratio Initial greed rate e0, final greed rate e f ; S2.4, initialize the population, screen the individuals in the population through the fitness function, and obtain the non-dominated solution set Π in the population; S2.5, Destruction-Reconstruction: Perform a destruction-reconstruction operation on the non-dominated solution set π in the population, that is, P Randomly select d products from the set, and completely destroy and reconstruct the workpiece of each product; find the union of the destroyed and reconstructed population and Π, and search for non-dominated solutions again to obtain a new non-dominated solution set Π; S2.6, energy-saving strategy: perform energy-saving operations on the non-dominated solution set Π; S2.7, generate high-level individuals according to the Q table, each high-level individual is composed of a low-level heuristic sequence; S2.

8. Apply each high-level individual to Π, calculate the contribution rate of each high-level individual, and select the elite high-level individuals. S2.9, apply the elite high-level individuals to Π, update the Q-table and non-dominated solution set Π; S2.

10. Determine whether the termination condition (iteration number = 50) is met. If so, output the non-dominated solution set Π; otherwise, jump to S2.

7.

3. The scheduling method according to claim 2, characterized in that: The specific encoding scheme is as follows: For EE-DAHFSP-PM, each feasible solution (π, V) is represented by a two-dimensional sequence π = {π F ,π P ,π J } and the velocity matrix V, the sequence π has three rows, π F Represents the factory sequence, which contains the factory allocation information for all products; π P It represents the processing order of all products; π J It contains the processing sequence information of all the workpieces belonging to the product; the speed matrix V contains the information of the processing speed of each workpiece on different machines during the production stage.

4. The scheduling method according to claim 2, characterized in that: The LLHs pool includes 12 low-level heuristics LLH1-LLH 12 , each of which is defined as a state S, and the operation taken by the state is called action a. In the Q table, the state s is consistent with the elements of action a, which consists of 12 low-level heuristics.

5. The scheduling method according to claim 2, characterized in that: Using ε T As the greedy probability in the greedy strategy, for a specific state, there is 1-ε T The probability of selecting the action with the largest Q value in the Q table, ε T The calculation formula is as follows: Where ε0 is the initial greed rate (ε0 = 0.15), ε f is the final greed rate (ε f =0.01), T total is the total number of iterations of the algorithm, T now The current iteration round.

6. The scheduling method according to claim 2, characterized in that: In S2.7, the specific steps of generating high-level individuals according to the Q table include: S2.7.

1. Select a state s according to the SelectAnAction algorithm t , get action a t and the next state s t+1 , and perform action a on the non-dominated solution set Π in the population t , obtain a new non-dominated solution set Π′ and evaluate the performance of Π′; The specific way to evaluate the new solution is as follows: during the operation of QLHHEA, the action selected by Q-learning is applied to the Pareto solution set; the value of the selected action is determined by the quality of the generated Pareto frontier (PF); the performance of the action is evaluated by the convergence index (CV) and the diversity index (DV), CV, DV are defined by the following formulas respectively: Among them, P t Indicates that by in state s t Next, perform action a t The obtained Pareto frontier (PF), P * represents the reference Pareto optimal frontier, d i Indicates P t The Euclidean distance between each pair of adjacent points in , represents the average of these distances, and the N variable represents P t The number of non-dominated solutions in the distance d f and d l Respectively represent P t The Euclidean distance between the extreme value solution and the boundary solution, the smaller the CV t The value indicates that P t Stronger convergence, larger DV t The value indicates that P t The diversity of the solution is stronger, so the following four performance evaluation (PE) indicators are defined: PE1: DV t <DV t+1 and CV t >CV t+1 ;PE2:DV t <DV t+1 and CV t ≤CV t+1 ;PE3:DV t ≥DV t+1 and CV t >CV t+1 ;PE4:DV t ≥DV t+1 and CV t ≤CV t+1 ; S2.7.2, set the reward function, update the Q table according to the reward r obtained, and change the state s t Update to the next state s t+1 , reward function r(s t ,a t ) and the Q table update formula are as follows: S2.7.

3. If all high-level individuals are generated, jump to S2.8, otherwise jump to S2.

7.

7. The scheduling method according to claim 4, characterized in that: 12 low-level heuristics LLH1-LLH 12 It includes: 9 product parts and 3 process parts; The 9 product sections include: LLH1: key products and non-key products are exchanged, LLH2: non-key products are inserted after key products, LLH3: key products and non-key products are reversed, LLH4: product exchange, LLH5: product front insertion 1, LLH6: product front insertion 2, LLH7: product reverse 1, LLH8: product reverse 2, LLH9: product reverse 3; The 3 process parts include: LLH 10 : Workpiece exchange, LLH 11 : Workpiece forward insertion 1, LLH 12 : Workpiece forward insertion 2.

Images