A distributed assembly flow shop scheduling method based on deep reinforcement learning

Abstract

This invention relates to a distributed assembly line scheduling method based on deep reinforcement learning, belonging to the field of assembly line scheduling. The invention includes obtaining parameters of the distributed assembly line scheduling problem; constructing a model of the distributed assembly line scheduling problem based on these parameters; the scheduling model of the distributed assembly line scheduling problem includes an objective function and constraints; and solving the distributed assembly line scheduling problem using a Q-Learning-based hyperheuristic algorithm. During the solution process, an optimal solution is obtained in each iteration and used as the initial solution for the next iteration. The final solution generated through continuous iteration is then considered the optimal solution. This invention obtains a high-quality solution to the distributed assembly line scheduling problem within a finite time, effectively improving enterprise production efficiency. The scheduling method performs excellently in large-scale production environments, helping manufacturing enterprises complete tasks faster and more efficiently.

Description

Technical Field

[0001] This invention relates to a distributed assembly line workshop scheduling method based on deep reinforcement learning, belonging to the field of assembly line workshop scheduling technology. Background Technology

[0002] Machinery manufacturing is the foundation of modern industry, playing a crucial role in improving production efficiency, reducing costs, enhancing product quality, and meeting market demands. With the continuous development and innovation of manufacturing technology, the machinery manufacturing industry has entered a new stage, in which distributed production assembly has become a topic of great interest.

[0003] In traditional machinery manufacturing, all production and assembly work is usually completed in a centralized factory. However, this centralized production model has many limitations. For example, if a company wants to expand its production scale, it must increase production equipment and personnel, which will lead to a continuous increase in production costs. For serving customers, since products must be transported from the factory to the place of consumption, this will increase transportation costs and time costs, and the products may also be damaged during transportation.

[0004] To address the aforementioned problems, distributed production and assembly manufacturing has emerged. Distributed production and assembly scheduling refers to dispersing the production process across different locations and utilizing network and information technology to achieve collaborative manufacturing. Through distributed production and assembly scheduling, enterprises can distribute production processes to different locations for parallel processing, thereby effectively shortening the manufacturing cycle, reducing production costs, and better meeting the market demands of different regions. For customers, since products can be produced closer to them, transportation and time costs are also significantly reduced.

[0005] With the continuous development and improvement of intelligent scheduling systems, distributed production and assembly scheduling has become a crucial issue that urgently needs to be addressed in the field of production scheduling. Through distributed production and assembly, enterprises can reduce production costs, improve production efficiency, and enhance product quality, while also better meeting market demands. Therefore, for the machinery manufacturing industry, mastering the optimization technologies and scheduling methods of distributed production and assembly is undoubtedly an important way to improve industry competitiveness.

[0006] The Distributed Assembly Permutation Flow-shop Scheduling Problem (DAPFSP) is one of the most widely studied scheduling problems. It is a simplified model of many practical flow-shop production scheduling problems, and its optimization objective is usually to minimize the total production time by minimizing the production time (makespan) of each factory. Pan et al. first studied the Distributed Assembly Permutation Flow-shop Scheduling Problem (DAPFSP) based on AFSP and DPFSP. DAPFSP involves multiple identical factories, each containing several processing machines and one assembly machine. In the real world, scheduling problems in various multi-factory collaborative manufacturing can be directly modeled as DAPFSP. A typical example is the manufacturing of automobile engines. Automobile engines generally contain various parts such as cylinder blocks, cylinder heads, bottom casings, and crankshafts. These parts are first manufactured in a flow-shop and then assembled into the finished engine in an assembly machine. Therefore, each factory is responsible for the processing and assembly of many parts. In actual production and manufacturing processes, due to the complexity of the products being processed, the numerous parts in each product, and the multiple processes involved in each processing plant, determining the allocation of tasks within the plant, the processing sequence of the products, the processing sequence of the parts in each product, and the final assembly sequence of the product, thereby minimizing the maximum completion time in the entire processing process, is a critical issue that urgently needs to be addressed.

[0007] There is a wealth of research on the distributed assembly displacement flow shop scheduling problem. Hatami et al. proposed three heuristic algorithms and a fast variable neighborhood descent algorithm to address this problem, evaluating the algorithms using 900 small-scale instances and 810 large-scale instances. In addition, Li et al. proposed a genetic algorithm (GA) employing an enhanced crossover operator and three local search strategies; Wang et al. proposed a memetic algorithm based on distribution estimation (EDAMA); Lin and Zhang proposed a hybrid optimization algorithm (HBBO) based on biogeographical theory; Lin et al. proposed a backtracking search hyperheuristic algorithm (BH-HH); and Zhang et al. proposed a matrix cube-based distribution estimation algorithm (MCEDA). These methods have effectively solved the problem to some extent, but most of them employ traditional metaheuristic strategies. Recent research trends indicate that hyperheuristic methods have become a more effective and efficient search method. Reinforcement learning, as an emerging artificial intelligence technology, is also beginning to be applied to solve the shop floor production scheduling problem. Reinforcement learning has developed rapidly in recent years, achieving many breakthroughs in multiple fields. Reinforcement learning has strong adaptability and can effectively solve dynamic scheduling problems. However, existing methods have not yet combined reinforcement learning techniques with hyperheuristic algorithms for solving distributed assembly line scheduling problems. Traditional methods are slow in solving scheduling models and consume more resources.

[0008] The inventors discovered that existing technologies suffer from the following technical bottlenecks:

[0009] First, most existing studies on distributed assembly line scheduling problems focus on parts being processed in multiple factories and then assembled on a single assembly machine. However, an increasing number of large and medium-sized companies own multiple assembly-oriented factories, which may be located in different parts of the world. For these companies, undertaking a large manufacturing project requires multiple factories to collaborate in production to ensure on-time delivery. Research on directly assembling products within the same factory is still relatively limited.

[0010] Second, most existing distributed assembly replacement flow shop scheduling methods employ traditional metaheuristic approaches, failing to effectively utilize global and local searches. The global and local searches are not yet coordinated, and the impact of different search orders on feasible solution sequences is not considered, leading to the neglect of many high-quality solutions during the search process. Furthermore, existing scheduling methods have not yet combined reinforcement learning techniques with metaheuristic algorithms to solve the distributed assembly replacement flow shop scheduling problem, resulting in slower model solving speeds and higher resource consumption. Summary of the Invention

[0011] To address the shortcomings of the existing technologies, this invention proposes a distributed assembly line scheduling method based on deep reinforcement learning, which improves the overall efficiency of the workshop scheduling and reduces the completion time.

[0012] The technical solution of this invention is: a distributed assembly line workshop scheduling method based on deep reinforcement learning, comprising:

[0013] Obtain parameters for the distributed assembly replacement flow workshop;

[0014] Based on the parameters of the distributed assembly replacement flow workshop, a model of the distributed assembly flow workshop is constructed; the scheduling model of the distributed assembly flow workshop includes: objective function and constraints.

[0015] Using a Q-Learning-based hyperheuristic algorithm, a distributed assembly displacement flow workshop is solved. In the process of solving, an optimal solution is obtained in each generation and used as the initial solution for the next iteration. After continuous iteration, the final solution is generated and used as the optimal solution.

[0016] Among them, the Q-Learning-based hyperheuristic algorithm refers to using the Q-Learning learning strategy as a high-level strategy (HLS) to guide the low-level heuristic algorithm, thereby forming a new heuristic algorithm (Low-Level Heuristics, LLHs) to search different regions of the solution space and improve the global search capability of the algorithm; the low-level heuristic algorithm searches for the product processing sequence through insertion and exchange operations between and within factories.

[0017] As a further aspect of the present invention, the acquisition of parameters for the distributed assembly and replacement flow workshop specifically includes:

[0018] The number of factories, the number of machines, the number of workpieces, the number of products, the standard processing time for each workpiece on the machine tool, the standard processing time for each product on the assembly machine tool, and the allocation of workpieces to products.

[0019] As a further aspect of the present invention, the model of the distributed assembly line workshop is constructed based on the parameters of the distributed assembly line workshop; the scheduling model of the distributed assembly line workshop includes: constraints, objective function, and encoding method.

[0020] The constraints are as follows: there are n workpieces and δ products, and the products are assembled from the workpieces; each workpiece belongs to one product.

[0021] The distributed assembly replacement flow shop scheduling problem includes two phases: the production phase and the assembly phase.

[0022] Each factory consists of several processing machines for product processing and one assembly machine for product assembly.

[0023] During the production phase, there are a set of f identical factories, each independent of the others. Each factory is a production line workshop consisting of m machines. The production phase consisting of m machines processes each part of the product like a production line workshop. Within each factory, all parts must be processed in the same processing sequence. All parts of each product must be processed continuously in the same factory. On each machine in each factory, only after all parts of one product have been processed can the processing of parts for the next product begin. Once the processing operation of each part starts, it cannot be interrupted.

[0024] During the assembly phase, an assembly machine M is used. A The assembly stage is responsible for assembling the manufactured parts into a product; the assembly of each product must be carried out on an assembly machine in the same factory where it was processed; if all parts of the product have been processed and the assembly machine is idle, the product assembly operation is allowed to begin immediately; once the product assembly begins, it must be carried out without interruption.

[0025] The objective function is to minimize the maximum completion time (makespan).

[0026] Here, encoding refers to the way the solution is represented. A solution is represented using a set of f sequences, one sequence for each factory. Each sequence is a permutation of all parts assigned to that factory, representing the order in which the parts enter the assembly line. Since it is restricted that all workpieces belonging to the same product will never be separated, the order in which products are processed on the assembly machine is also implicitly included. Using this representation, the feasible scheduling solution is represented as π = {π1, π2, ..., π1}. f},in It is the sequence of workpieces in factory k, n k It is the total number of parts allocated to factory k.

[0027] As a further aspect of the present invention, the distributed assembly line replacement workshop is solved using a Q-Learning-based hyperheuristic algorithm. During the solution process, an optimal solution is obtained in each generation and used as the initial solution for the next iteration. The final solution generated through continuous iteration is then considered the optimal solution. Specifically, this includes:

[0028] Step 1: Determine the encoding and decoding methods;

[0029] Step 2: Using the efficient Heu method, construct the initial work sequence for each factory and initialize the Q-table for each state-action pair;

[0030] Step 3: Evaluate the fitness of the solution through the decoding scheme and save the initial solution as the global optimal solution;

[0031] Step 4: Design low-level heuristic operations; Based on the exchange, insertion, and reversal operations of processed products and parts, eight low-level heuristic operations were designed, and an action selection strategy was used to select action a; The action selection strategy (i.e., LLH selection) retains a certain degree of exploration in the early stage of action selection, and as the number of training times increases, it tends to use the learned knowledge to guide the selection of actions.

[0032] Step 5: Execute the selected operation 'a' from the action set to search the solution space and evaluate the new solution; when a better solution is found, update the global optimal solution; otherwise, use the simulated annealing algorithm with a constant temperature parameter T to determine whether to accept the current solution. If accepted, retain the solution and use it as the initial solution for the next iteration to search the solution space.

[0033] Step 6: Update the Q-table according to the Q-function formula; the continuous updating of the Q-table ensures that the algorithm always searches in a favorable direction; the learning rate determines the algorithm's ability to accept new information or retain existing information. In the early iterations of the Q-learning algorithm, the current solution is often not the optimal solution. As the algorithm's running time increases, the obtained solution gradually approaches the optimal solution.

[0034] Step 7: Check if the algorithm meets the stopping condition. If not, return to Step 3; otherwise, save the optimal solution π. best ;

[0035] Step 8: Output the optimal scheduling solution π best The corresponding scheduling Gantt chart is used to optimize the solution of the distributed assembly replacement flow shop scheduling problem.

[0036] As a further aspect of the present invention, the efficient Heu method is used to construct the initial work sequence for each factory, specifically:

[0037] First, an initial part sequence is generated for each product by sorting its parts in descending order of total processing time. Next, the NEH algorithm is used to sort the parts of each product, generating the optimal part sequence for each product. Then, an initial product sequence is generated by descending the completion time of all products. Finally, products are sequentially removed from the product sequence from the first to the last. The corresponding removed product sequence is then inserted into each position in the scheduled sequence of all factories, while simultaneously inserting each part of the product sequentially according to the NEH algorithm. The position that minimizes the maximum completion time of the partial sequence is selected as the final insertion position for the product.

[0038] As a further aspect of the present invention, the action selection strategy is used to select action a;

[0039] The actions are constructed from eight simple low-level heuristics, defined as follows:

[0040] LLH1: Remove each product from the critical plant and insert it into all possible locations within the original plant; select the location with the best makespan.

[0041] LLH2: Remove each product from the critical plant and swap it with all possible locations within that plant; select the location with the best makespan.

[0042] LLH3: Remove each product from the critical plant and insert it into all possible locations in all other plants; select the location with the best makespan.

[0043] LLH4: Remove every product from the critical plant and swap it with all possible locations in all other plants; select the location with the best makespan.

[0044] LLH5: Randomly remove 2 products from all factories and insert them into all possible positions in all factories; select the position with the best makespan.

[0045] LLH6: Randomly remove 3 products from all factories and insert them into all possible positions in all factories; select the position with the best makespan.

[0046] LLH7: Remove each workpiece from the critical plant and swap it with all possible workpiece locations for that product; select the location with the best makespan.

[0047] LLH8: Remove each workpiece from the critical plant and insert it sequentially into all possible locations for that product; select the location with the best makespan.

[0048] The beneficial effects of this invention are:

[0049] (1) With the goal of minimizing the maximum completion time, this invention designs a super-heuristic algorithm based on the Q-Learning algorithm framework. This invention designs a vector encoding mechanism based on the selection of the factory, the sorting of products and the sorting of workpieces, and proposes a heuristic method that considers the sorting of products and workpieces at the same time.

[0050] (2) To enhance the algorithm's local search capability, this invention designs eight different low-level heuristic operations. Then, to effectively guide the search direction, this invention utilizes the Q-Learning algorithm to select the optimal low-level heuristic operation through an evolutionary process and execute the search. This high-level strategy improves the efficiency and accuracy of the search. Furthermore, to prevent the algorithm from getting trapped in local optima, this invention also introduces a simulated annealing algorithm to guide the algorithm in exploring more promising regions within the search space. Finally, through iterative optimization, a final scheduling scheme that satisfies the requirements of mechanical equipment production and assembly is obtained.

[0051] (3) This invention is the first to use the Q-Learning algorithm of reinforcement learning to solve the distributed permutation flow shop scheduling problem. Based on the characteristics of the problem, a Q-table update mechanism is designed to increase the pertinence of the global algorithm and improve the global search capability.

[0052] (4) This invention proposes a neighborhood search method for workpiece sequence, which further improves the local search capability. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the DAPFSP of the present invention;

[0054] Figure 2 This is a schematic Gantt chart of the DAPFSP of the present invention;

[0055] Figure 3 This is a flowchart of the algorithm of the present invention. Detailed Implementation

[0056] Example 1: To solve the scheduling problem of distributed assembly and replacement flow workshop, Figures 1-3 As shown, the present invention provides a distributed assembly line workshop scheduling method based on deep reinforcement learning, including: obtaining parameters of the distributed assembly replacement line workshop.

[0057] Based on the parameters of the distributed assembly replacement flow workshop, a model of the distributed assembly flow workshop is constructed; the scheduling model of the distributed assembly flow workshop includes: objective function and constraints.

[0058] Using a Q-Learning-based hyperheuristic algorithm, a distributed assembly displacement flow workshop is solved. In the process of solving, an optimal solution is obtained in each generation and used as the initial solution for the next iteration. After continuous iteration, the final solution is generated and used as the optimal solution.

[0059] Among them, the Q-Learning-based hyperheuristic algorithm refers to using the Q-Learning learning strategy as a high-level strategy (HLS) to guide the low-level heuristic algorithm (LLHs), thereby forming a new heuristic algorithm to search different regions of the solution space and improve the global search capability of the algorithm. The low-level heuristic algorithm searches for the product processing sequence through operations such as insertion and exchange between and within factories.

[0060] Obtain the parameters for the distributed assembly replacement flow shop scheduling method; specifically including:

[0061] The number of factories, the number of machines, the number of workpieces, the number of products, the standard processing time for each workpiece on the machine tool, the standard processing time for each product on the assembly machine tool, and the allocation of workpieces to products.

[0062] Based on the parameters of the distributed assembly line workshop, a model of the distributed assembly line workshop is constructed; the scheduling model of the distributed assembly line workshop includes: constraints, objective function, and encoding method.

[0063] The constraints are as follows: there are n workpieces and δ products, and the products are assembled from the workpieces; each workpiece belongs to one product.

[0064] Specifically, this invention also includes (1) establishing a mathematical model of DAPFSP; (2) designing an encoding method based on the selection of factories, the sorting of products and the sorting of workpieces, and proposing a heuristic method that considers both product and workpiece sorting simultaneously; (3) in order to improve the local search capability of the algorithm, this invention designs a set of eight heuristic operations based on different methods such as swapping, insertion, and reversal as alternative actions for the Q-Learning algorithm; (4) in order to effectively guide the search direction, the Q-Learning algorithm is used to select the best low-level heuristic operation through the evolutionary process and perform the search; (5) in order to avoid the algorithm getting trapped in local optima, a simulated annealing algorithm is introduced to guide the algorithm to explore more promising regions in the search space.

[0065] A. Symbols and Explanations

[0066] (1) Symbol

[0067] Table 1. Parameter and Symbol Representation

[0068]

[0069]

[0070] (2) Problem Description

[0071] The DAPFSP problem is described as follows: There are f identical factories, each consisting of several processing machines for product manufacturing and one assembly machine for product assembly. A production process consisting of m machines M = {M1, M2, M3, ..., M4} processes each part of the product like an assembly line. The assembly machine M... A The processing stage is responsible for processing the manufactured parts into finished products. For a set of δ products, Δ={Δ1,Δ2,Δ3,...,Δ...} δ}, each product Δ l l = 1, 2, 3, ..., δ are all determined by n l Each product consists of several workpieces. Each product can be assigned to any factory because they are identical. However, all parts of a product must be processed continuously within the same factory; that is, only after all parts of one product have been processed on each machine in each factory can the parts of the next product be processed. Each factory is a relatively independent assembly line workshop, and all parts in the factory must be processed in the same order: first on machine M1, then on machine M2, and so on, until M... m Machine. In machine M i The product Δ l The operation of machining the j-th part is represented as O. l,j,i O l,j,i The processing time is denoted as P. l,j,i Product Δ l Once each part is machined, it is immediately delivered to the assembly machine in the same factory. If all parts for the product have arrived and the assembly machine is available, the product assembly operation begins immediately. Product Δ l In assembly machine M A The operation time on is represented as O l O l The processing time is expressed as P. l We assume that P for all factories l,j,i and P l They are all the same. Whether it's processing operation O l,j,i Or is it assembly operation O? lOnce the operation is initiated, it must be carried out without interruption. The next issue is to determine the allocation of products to the various factories, and the order in which parts and products are allocated to each factory, in order to minimize the maximum completion time for all products.

[0072] (3) Constraints

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] c j,i ≥c j,i-1 +p j,i i=1,2,...,m,j=1,2,...,n (7)

[0080]

[0081]

[0082] c l ≥c j,m +p l j=z l-1 +1,...,z l ,l=1,2,...,δ (10)

[0083] c l ≥c l' +p l +(y l',l -1)·h,l=1,2,...,δ,l′=0,1,2,...,δ,l≠l′ (11)

[0084]

[0085] x j,j' ∈{0,1},j=0,1,2,...,n,j′=0,1,2,...,n,j≠j′ (13)

[0086] Where c j，0 =0, c 0,i =0, c 0,i=0. Equations (1) and (2) ensure that each part must have exactly one direct predecessor and one direct successor. Equations (3) and (4) force that the predecessor and successor of all factory products must be virtual parts 0. Equation (5) ensures that no part will be repeated and will only be processed once. Equation (6) ensures that parts belonging to the same product will never be separated or mixed with parts belonging to other products. Equation (7) represents operation O. j,i Only in its preceding operation O j,i-1 Only after completion can it begin. Equation (8) determines if part ω j′ It is part of ω j The direct predecessor, then the operation of O j,i Only when operating O j',i After completion, it begins. Equation (9) indicates that the processing sequence relationship between two products is determined by the processing sequence relationship of their parts. Equation (10) means that the assembly operation cannot begin until all parts of the product are completed. Equation (11) indicates that if product Δ l′ Placed on product Δ l Previously, the product Δ l Assembly must be performed on the product Δ l′ The assembly operation can only begin after the assembly operation is completed. Equation (12) defines the completion time. Equation (13) defines the range of the decision variables.

[0087] B. Optimize the scheduling model and optimization objectives

[0088] Based on the constraints of DAPFSP and the order of parts processing, product processing, and factory allocation, DAPFSP can be divided into two stages: (1) the production stage and (2) the assembly stage. Therefore, an optimization scheduling model for the processing and assembly stages can be established:

[0089] (1) In the production stage, there are a set of f identical factories that are independent of each other. Each factory is a production line workshop consisting of m machines. The production stage consisting of m machines processes each part of the product like a production line workshop. In each factory, all parts must be processed in the same processing order. All parts of each product must be processed continuously in the same factory. Only after all parts of a product have been processed on each machine in each factory can the parts of the next product be processed. Once the processing operation of each part begins, it cannot be interrupted.

[0090] Depending on the processing sequence, the processing time of a part can be calculated in four ways:

[0091] 1) First, in the production stage, the completion time of the first workpiece in each factory on the first machine is as follows: According to the analysis, the start time of processing the first workpiece in each factory on the first machine is 0, and the completion time is equal to the processing time required for that workpiece. The corresponding calculation is as follows:

[0092]

[0093] in This represents the first workpiece in the workpiece sequence of the f-th factory. This represents the processing time of the first workpiece in the workpiece sequence of the f-th factory on the first machine. This represents the completion time of the first workpiece in the workpiece sequence of the f-th factory on the first machine;

[0094] 2) Next is the completion time of the i-th workpiece in each factory on the first machine during the production stage. Analysis shows that each part is processed continuously on the first machine in each factory, that is, the start time of processing is equal to the completion time of the previous part. The corresponding calculation formula is as follows:

[0095]

[0096] in This represents the i-th workpiece in the workpiece sequence of the f-th factory. Let represent the processing time of the i-th workpiece in the workpiece sequence of the f-th factory on the first machine. Let represent the completion time of the i-th workpiece in the workpiece sequence of the f-th factory on the first machine; during the production stage, the completion time of the i-th workpiece in each factory on the first machine is equal to the sum of the completion time of the (i-1)-th workpiece in each factory on the first machine and the processing time of the i-th workpiece on the first machine.

[0097] 3) Next is the completion time of the first workpiece in each factory on the j-th machine during the production stage. Analysis shows that the start time of processing the first workpiece in each factory on the j-th machine depends solely on the completion time of that part on the previous machine. The corresponding calculation formula is as follows:

[0098]

[0099] in Let represent the processing time of the first workpiece in the workpiece sequence of the f-th factory on the j-th machine (j≥2). Let represent the completion time of the i-th workpiece in the workpiece sequence of the f-th factory on the first machine; the completion time of the first workpiece in each factory on the j-th machine during the production stage is equal to the sum of the completion time of the first workpiece in each factory on the (j-1)-th machine and the processing time of the workpiece on the j-th machine.

[0100] 4) Finally, the completion time of the i-th workpiece in each factory on the j-th machine during the production stage is analyzed. It can be seen that the start time of all other parts is equal to the maximum of the completion time of the previous part on that machine and the completion time of that part on the previous machine. The calculation is as follows:

[0101]

[0102] In the production stage, the completion time of the i-th workpiece in each factory on the j-th machine is equal to the maximum of the completion time of the i-th workpiece on the j-1-th machine and the completion time of the i-1-th workpiece on the j-th machine, plus the processing time of the workpiece on the j-th machine.

[0103] (2) During the assembly stage, an assembly machine M is used. A The assembly stage is responsible for assembling the manufactured parts into a product; the assembly of each product must be carried out on an assembly machine in the same factory where it was processed; if all parts of the product have been processed and the assembly machine is idle, the product assembly operation is allowed to begin immediately; once the product assembly begins, it must be carried out without interruption.

[0104] During the assembly stage, each finished part is assembled into the final product. Analysis shows that the start time of the first product in each factory on the assembly machine depends on the time it takes for the last part of that product to be completed. The specific calculation formula is as follows:

[0105]

[0106] The start-up time for the l-th product (l≠1) in each factory on the assembly machine depends on the maximum of the time it takes for the last part of that product to be completed and the completion time of the previous product on the assembly machine in that factory. The specific calculation formula is as follows:

[0107]

[0108] in Let represent the completion time of the l-th product in factory f on the assembly machine. Due to the constraint that all workpieces of the same product must be processed continuously in the same factory, the workpiece processing sequence of factory f is denoted as . The information also includes the product processing sequence of factory f; Represents a sequenceπf The number of parts in the first l products, i.e., the number of products in the first l products of factory f, where πf represents the product sequence at factory f. This represents the number of parts for the t-th product in factory f; during the assembly stage, the completion time of the first product in each factory is equal to the completion time of the last part processing stage of the first product plus the assembly time required for the first product; during the assembly stage, the completion time of the l-th product in each factory is equal to the maximum value of the completion time of the (l-1)-th product in that factory and the completion time of the last part processing stage of the l-th product, plus the assembly time required for the l-th product.

[0109] Finally, the formula for calculating the optimization target's maximum completion time (makespan) can be obtained. This equals the maximum time required to assemble all products in each factory, as shown in the following formula:

[0110]

[0111] C. Model Method

[0112] The overall process for solving the distributed assembly displacement flow shop scheduling problem, where the objective function is to minimize the maximum completion time, is as follows: Figure 3 As shown, the specific implementation steps are as follows:

[0113] Step 1: Determine the encoding and decoding methods.

[0114] In the encoding process, this invention uses a set of f sequences to represent a solution, one sequence for each factory. Each sequence is a permutation of all parts assigned to that factory, representing the order in which the parts enter the assembly line. Since it is restricted that all workpieces belonging to the same product will never be separated, the order in which products are processed on the assembly machine is also implicitly included. Using this representation, a feasible scheduling solution can be expressed as π = {π1, π2, ..., π3}. f},in It is the sequence of workpieces in factory k, n k It is the total number of parts allocated to factory k;

[0115] To make it easier to understand, let's take a simple example. Suppose there are 4 products, 8 parts, and 2 factories. Each factory has 2 identical processing machines and 1 assembly machine. The correspondence between products and parts is as follows: Λ1 = {1}, Λ2 = {2,3}, Λ3 = {4,5}, Λ4 = {6,7,8}.

[0116] Table 2 shows the processing time for parts and products in this example, and one of the corresponding feasible codes is as follows:

[0117] π = {{2,3,1},{7,8,6,5,4}}, where π1 = {2,3,1} represents the processing sequence of the parts in the first factory. Since all parts of a product must be processed continuously, this coding method also implies product coding information.

[0118] Table 2 Example of Processing Data

[0119]

[0120] Decoding refers to determining the maximum completion time (makespan) of the scheduling scheme by determining the product allocation of each factory, the processing sequence of products in each factory, and the processing sequence of parts within each product. The decoding strategy of this invention is divided into a parts processing stage and a product assembly stage. The parts processing stage determines the start and finish times of each part's processing. The product assembly stage determines the start time of product assembly by using the finish time of the last part of each product in the first stage and the finish time of the previous product on the factory's assembly machine. This process is then used to determine the completion time of each factory sequentially, ultimately obtaining the optimized maximum completion time (makespan). Detailed decoding strategies can be found in the optimization scheduling model and optimization objective section above.

[0121] Step 2: Using the efficient Heu method, the initial job sequence for each factory was constructed. The Q-table for each state-action pair was initialized;

[0122] The specific construction method of efficient Heu is as follows:

[0123] First, an initial part sequence is generated for each product by sorting its parts in descending order of total processing time. Next, the NEH algorithm is used to sort the parts of each product, generating the optimal part sequence for each product. Then, an initial product sequence is generated by descending the completion time of all products. Finally, products are sequentially removed from the product sequence from the first to the last. The test inserts the corresponding removed product sequence into every position in the scheduled sequence of all factories, while simultaneously inserting each part of the product sequentially according to the NEH algorithm. The position that minimizes the maximum completion time of the partial sequence is selected as the final insertion position for the product. Details of the efficient Heu algorithm are as follows.

[0124] Step 1: For product Δ l First, for each product, according to each part Λ l The total processing time is used to sort all parts in descending order to determine the part sequence S. l Then, the NEH method was used to improve each S. l And calculate its completion time.

[0125] Step 2: Obtain the product sequence S by sorting all products in descending order of maximum completion time.

[0126] Step 3: Take the first f products from sequence S and assign one to each factory. For each factory, set π. k As a sequence of parts for a specified product.

[0127] Step 4: If S is not empty, take the first product, let it be Δ. l Perform steps 4.1 and 4.2.

[0128] Step 4.1: For k = 1, 2, ..., f, try to achieve the following without disrupting the internal workpiece sequence of other products in π. k Insert S sequentially at each possible insertion position using the NEH method. l Each workpiece. Using C' k Let l' represent the maximum completion time, and the corresponding insertion position is l'. k The corresponding workpiece sequence is π' l .

[0129] Step 4.2: Find the factory k* with the minimum maximum completion time, i.e., k* = arg(min k＝1,...,f (C' k )). Put Δ l Assign it to factory k*, and put S l According to the workpiece sequence π' l Insert into π k* l' on k* Location.

[0130] Step 5: Repeat step 4 until all products have been taken into account.

[0131] Compared with methods that generate solutions randomly, the efficient Heu method can effectively improve the quality of the initial solution.

[0132] Step 3: Evaluate the fitness of the solution through the decoding scheme and save the initial solution as the global optimal solution.

[0133] Step 4: Design low-level heuristic operations. Eight low-level heuristic operations were designed based on operations such as exchanging, inserting, and reversing products and parts, and an action selection strategy was used to select action a. The action selection strategy (i.e., LLH selection) retains a certain degree of exploration in the early stage of action selection, and as the number of training sessions increases, it tends to use the learned knowledge to guide the selection of actions.

[0134] The eight simple low-level heuristic constructions are as follows:

[0135] LLH1: Remove each product from the critical plant and insert it into all possible locations within the original plant. Select the location with the best makespan.

[0136] LLH2: Remove each product from the critical plant and swap it with all possible locations within that plant. Choose the location with the best makeup span.

[0137] LLH3: Remove each product from the critical plant and insert it into all possible locations in all other plants. Choose the location with the best makespan.

[0138] LLH4: Remove every product from the critical plant and swap it with all possible locations in all other plants. Choose the location with the best makespan.

[0139] LLH5: Randomly remove 2 products from all factories and insert them into all possible positions in all factories. Select the position with the best makespan.

[0140] LLH6: Randomly remove 3 products from all factories and insert them sequentially into all possible positions in all factories. Select the position with the best makespan.

[0141] LLH7: Remove each workpiece from the critical plant and swap it with all possible workpiece locations for that product. Select the location with the best makespan.

[0142] LLH8: Remove each workpiece from the critical plant and insert it sequentially into all possible locations for that product. Select the location with the optimal makespan.

[0143] The action selection strategy is as follows:

[0144]

[0145]

[0146] Where ε is a value that follows a standard normal distribution over time t; t max The stopping criterion is the program's set running time. The formula for calculating ε indicates that at the start of training, the probability of exploring new actions is close to 50%, but as time t increases, the algorithm tends to select actions with the largest q value. In other words, the algorithm's action selection strategy (i.e., LLH selection) retains a certain degree of exploration in the early stages of action selection, and as the number of training iterations increases, it tends to use learned knowledge to guide action selection.

[0147] Step 5: Execute the selected operation 'a' from the action set to search the solution space and evaluate new solutions. When a better solution is found, update the global optimum. Otherwise, use simulated annealing with a constant temperature parameter T to determine whether to accept the current solution. If accepted, retain this solution and use it as the initial solution for the next iteration to search the solution space.

[0148]

[0149] β is a parameter set in the algorithm. n is the total number of parts, m is the number of machines, and p... l p is the processing time for the l-th product. l,j,i It is the processing time of the j-th workpiece of the l-th product on the i-th machine.

[0150] After searching and determining whether to retain a solution, if the current search round ends, the state is changed from S. t Updated to S t+1 The corresponding return rt can be obtained through the formula:

[0151] The reward and punishment mechanism used in the algorithm is represented as follows:

[0152]

[0153] Where u is a random number in [0,1], and r t Depends on ε t The probability and the aggregated state after the event ends.

[0154] Since different problems have different ranges of solution fitness, a fitness ratio P is used. t To separate and improve the space. P t The calculation is shown in the equation:

[0155]

[0156] in and These are the fitness values ​​of the solutions searched before and after within an iteration cycle EP, respectively, where t is the current event.

[0157] As can be seen from the formula, using different low-level heuristics will bring different rewards each time. If a better solution is obtained, the reward will be better, and vice versa.

[0158] Step 6: Update the Q table according to the Q function formula:

[0159]

[0160] The key parameters included in the Q function are the discount factor γ and the learning rate αt. t and reinforcement signal rt .

[0161] The discount factor γ represents the degree of influence of future rewards on the current situation, and its value is highly dependent on the problem domain.

[0162] Learning rate α t It represents the probability of retaining existing information or accepting new information. In this patent, α t The value decreases adaptively over time and is calculated according to formula (27).

[0163]

[0164] Where t is the current evaluation number, t max It is the total evaluation time, which is the program's set running time.

[0165] The continuous updating of the Q-table ensures that the algorithm always searches in a more favorable direction. The learning rate determines the algorithm's ability to accept new information or retain existing information. In the early iterations of the Q-learning algorithm, the current solution is often not optimal; as the algorithm's running time increases, the obtained solution gradually approaches the optimal solution.

[0166] Step 7: Check if the algorithm meets the stopping condition. If not, return to Step 3; otherwise, save the optimal solution π. best .

[0167] Step 8: Output the optimal scheduling solution π best The corresponding scheduling Gantt chart is used to optimize the solution of the distributed assembly replacement flow shop scheduling problem.

[0168] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A distributed assembly line workshop scheduling method based on deep reinforcement learning, characterized in that, include: Obtain parameters for the distributed assembly replacement flow workshop; Based on the parameters of the distributed assembly replacement flow workshop, a model of the distributed assembly flow workshop is constructed. The scheduling model of the distributed assembly line workshop includes: objective function and constraints; A Q-Learning-based hyperheuristic algorithm is used to solve a distributed assembly displacement flow shop problem. In each iteration, an optimal solution is obtained and used as the initial solution for the next iteration. This iterative process continues until a final solution is generated, which is then considered the optimal solution. Specifically, this includes: Step 1: Determine the encoding and decoding methods; Step 2: Using the efficient Heu method, construct the initial work sequence for each factory and initialize the Q-table for each state-action pair; Step 3: Evaluate the fitness of the solution through the decoding scheme and save the initial solution as the global optimal solution; Step 4: Design low-level heuristic operations; eight low-level heuristic operations were designed based on the exchange, insertion, and reversal operations of processed products and parts, and an action selection strategy was used to select the actions. The action selection strategy retains a certain degree of exploration in the early stages of action selection, and as the number of training sessions increases, it tends to use learned knowledge to guide action selection; the action selection strategy is the selection of LLH. Step 5: Perform the action set selection operation. The algorithm searches the solution space and evaluates new solutions. When a better solution is found, the global optimum is updated. Otherwise, a simulated annealing algorithm with a constant temperature parameter T is used to determine whether to accept the current solution. If accepted, the solution is retained and used as the initial solution in the next iteration to search the solution space. Step 6: Update the Q-table according to the Q-function formula; the continuous updating of the Q-table ensures that the algorithm always searches in a favorable direction; the learning rate determines the algorithm's ability to accept new information or retain existing information. In the early iterations of the Q-learning algorithm, the current solution is often not the optimal solution. As the algorithm's running time increases, the obtained solution gradually approaches the optimal solution. Step 7: Check if the algorithm meets the stopping condition. If not, return to Step 3; otherwise, save the optimal solution. ; Step 8: Output the optimal scheduling solution The corresponding scheduling Gantt chart is used to optimize the solution of the distributed assembly replacement flow shop scheduling problem. Among them, the Q-Learning-based hyperheuristic algorithm refers to using the Q-Learning learning strategy as a high-level strategy to guide the low-level heuristic algorithm, thereby forming a new heuristic algorithm to achieve the search of different regions of the solution space and improve the global search capability of the algorithm; the low-level heuristic algorithm achieves the search of the product processing sequence through insertion and exchange operations between and within factories.

2. The distributed assembly line scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The acquisition of parameters for the distributed assembly and replacement flow workshop specifically includes: The number of factories, the number of machines, the number of workpieces, the number of products, the standard processing time for each workpiece on the machine tool, the standard processing time for each product on the assembly machine tool, and the allocation of workpieces to products.

3. The distributed assembly line scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The model of the distributed assembly line workshop is constructed based on the parameters of the distributed assembly line workshop; the scheduling model of the distributed assembly line workshop includes: constraints, objective function, and encoding method. Among them, the constraint condition refers to: there are One workpiece and A product consisting of workpieces assembled from individual workpieces; each workpiece belongs to one product. The distributed assembly replacement flow shop scheduling problem includes two phases: the production phase and the assembly phase. Each factory consists of several processing machines for product processing and one assembly machine for product assembly. During the production phase, there is a set of completely identical [materials / products]. There are several factories, each independent of the others. Each factory is a production line workshop consisting of m machines. The production stages, composed of m machines, process each part of a product like an assembly line. Within each factory, all parts must be processed in the same order. All parts of each product must be processed continuously in the same factory. On each machine in each factory, the processing of parts for the next product can only begin after all parts of the previous product have been processed. Once the processing of a part begins, it cannot be interrupted. During the assembly phase, an assembly machine M is used. A The assembly stage is responsible for assembling the manufactured parts into a product; the assembly of each product must be carried out on an assembly machine in the same factory where it was processed; if all parts of the product have been processed and the assembly machine is idle, the product assembly operation is allowed to begin immediately; once the product assembly begins, it must be carried out without interruption. The objective function is to minimize the maximum completion time (makespan). Here, encoding refers to the way the solution is represented. A solution is represented using a set of f sequences, one sequence for each factory. Each sequence is a permutation of all parts assigned to that factory, representing the order in which the parts enter the assembly line. Since it is restricted that all workpieces belonging to the same product will never be separated, the order in which products are processed on the assembly machine is also implicitly included. Using this representation, the feasible scheduling solution is expressed as follows: ,in It is the sequence of workpieces in factory k. It is the total number of parts allocated to factory k.

4. The distributed assembly line scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The efficient Heu method is used to construct the initial work sequence for each factory, specifically: First, an initial part sequence is generated for each product by sorting its parts in descending order of total processing time. Next, the NEH algorithm is used to sort the parts of each product, generating the optimal part sequence for each product. Then, an initial product sequence is generated by descending the completion time of all products. Finally, products are sequentially removed from the product sequence from the first to the last. The corresponding removed product sequence is then inserted into each position in the scheduled sequence of all factories, while simultaneously inserting each part of the product sequentially according to the NEH algorithm. The position that minimizes the maximum completion time of the partial sequence is selected as the final insertion position for the product.

5. The distributed assembly line scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The action selection strategy is used to select actions. ; The actions are constructed from eight simple low-level heuristics, defined as follows: LLH1: Remove each product from the critical plant and insert it into all possible locations within the original plant; select the location with the best makespan. LLH2: Remove each product from the critical plant and swap it with all possible locations within that plant; select the location with the best makespan. LLH3: Remove each product from the critical plant and insert it into all possible locations in all other plants; select the location with the best makespan. LLH4: Remove every product from the critical plant and swap it with all possible locations in all other plants; select the location with the best makespan. LLH5: Randomly remove 2 products from all factories and insert them into all possible positions in all factories; select the position with the best makespan. LLH6: Randomly remove 3 products from all factories and insert them into all possible positions in all factories; select the position with the best makespan. LLH7: Remove each workpiece from the critical plant and swap it with all possible workpiece locations for that product; select the location with the best makespan. LLH8: Remove each workpiece from the critical plant and insert it sequentially into all possible locations for that product; select the location with the best makespan.

Images