Reinforcement learning driven distributed flexible job shop scheduling optimization system
By using a reinforcement learning-driven distributed flexible job shop scheduling optimization system, which employs multi-dimensional heuristic initialization, neighborhood search, and heuristic move acceptance criteria, the system solves the problem of solution space exploration and energy consumption in multi-objective optimization of DFJSP, and achieves efficient scheduling optimization.
Patent Information
- Application Number
- CN202410934721.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-22
AI Technical Summary
The existing distributed flexible job shop scheduling problem (DFJSP) is difficult to find the optimal solution within a reasonable time due to the dynamic nature of the manufacturing environment, the heterogeneity of tasks, and multi-objective constraints. Furthermore, traditional methods are not effective in multi-objective optimization.
A distributed flexible job shop scheduling optimization system driven by reinforcement learning is proposed, which includes a scheduling sequence initialization module, a Q-learning-based hyperheuristic framework module, a problem feature-based energy-saving strategy module, and a heuristic move acceptance criterion module. Through multi-dimensional heuristic initialization, neighborhood search operator, energy-saving operation, and heuristic move acceptance criterion, low-level heuristic methods are dynamically selected to balance solution space exploration and development.
It improves the solution quality and efficiency of the distributed flexible job shop scheduling problem, reduces energy consumption, avoids the algorithm getting trapped in local optima, and is suitable for multi-objective optimization.
Smart Images

Figure CN119002408A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of distributed production scheduling in manufacturing industry, and in particular to a distributed flexible job shop scheduling optimization system driven by reinforcement learning. Background Art
[0002] Economic globalization has brought a huge impact on traditional manufacturing. Manufacturing companies must break the traditional manufacturing model in order to achieve sustainable economic development and gain a foothold in the increasingly fierce global market. From the original single-factory manufacturing to multi-factory collaborative production. Distributed production environments are becoming increasingly important because they can provide companies with higher quality, shorter delivery time and lower cost products. Production scheduling is an integral part of the manufacturing system and directly affects the efficiency and competitiveness of enterprises. In particular, the distributed flexible job scheduling problem, which has important applications in the manufacturing process of enterprises, has attracted the attention of many researchers. The Distributed Flexible Job Shop Scheduling Problem (DFJSP), which focuses on improving energy efficiency and sustainability, has become an important research area in modern manufacturing systems. This problem summarizes the challenges of optimizing production plans at multiple distributed manufacturing sites, while taking into account the flexible allocation of jobs to machines to minimize production cycles, energy consumption and other goals. The global promotion of green manufacturing practices aims to reduce carbon footprint and operating costs while maintaining high production efficiency, which has driven the urgent need for energy-efficient solutions. Despite significant progress, DFJSP remains a complex problem due to the dynamic nature of manufacturing environments, the heterogeneity of tasks, and the presence of conflicting multi-objective constraints.
[0003] In DFJSP, three tasks need to be solved, namely, allocating workpieces to plants, confirming processes, and allocating processes to machines. Since the flexible job shop scheduling problem is already an NP-hard problem, and DFJSP is a special case of FJSP with multiple plants, it is also an NP-hard problem. At present, many optimization methods have been developed by researchers to solve DFJSP, including exact algorithms, heuristic methods, and meta-heuristic methods, such as genetic algorithms (GAs), particle swarm optimization (PSO), and artificial bee colonies (ABCs). When solving DFJSP, although meta-heuristic methods are expected to find near-optimal solutions in a reasonable time, meta-heuristic methods usually require a lot of parameter adjustments and may not be able to consistently obtain optimal solutions in different problem instances. In addition, these methods usually focus on single-objective optimization and may not be able to effectively solve DFJSP under multiple optimization objectives. In this case, the weights between total completion time, energy consumption, and other objectives must be balanced.
[0004] The selection hyperheuristic (SHH) is a flexible and powerful method for solving complex optimization problems. It can intelligently select and apply a set of low-level heuristic methods according to the state of the environment, and dynamically adjust according to the characteristics of different problems, thereby overcoming the limitations of traditional metaheuristic methods. Recent studies have begun to explore the application of selection hyperheuristics to various scheduling and optimization problems, and have demonstrated its high efficiency in exploring the solution space of complex optimization problems. In the selection hyperheuristic algorithm, the efficiency of the high-level strategy has a significant impact on the performance of the algorithm. How to design efficient high-level strategies has become a hot topic in the current research of hyperheuristic algorithms. As a branch of reinforcement learning algorithms, the Q-learning algorithm has achieved a series of research results in solving scheduling problems. Reinforcement learning serves the hyperheuristic algorithm, extracts knowledge from the feedback information generated during the entire search process, and reasonably integrates the knowledge into the search process at different stages of population evolution. It can guide the hyperheuristic algorithm to select the most appropriate low-level heuristics, fully explore and develop the solution space, and effectively improve the quality of the solution. The Q-learning-guided SHH uses the intelligent iterative mechanism of "feedback-training-selection" to train a self-learning agent based on problem knowledge. The agent intelligently selects low-level heuristics to obtain high-quality solutions, and then uses feedback information to learn and update the agent, thereby continuing to improve the efficiency of agent selection. The mobile acceptance criterion accepts new solutions (including poor solutions) with probability to ensure the diversity of solutions. The high-level strategies, low-level heuristics, and mobile acceptance criteria in SHH can be implemented and replaced in different ways according to the different needs of actual problems. It is a solution framework with high solution efficiency and strong versatility. Therefore, using SHH to solve the distributed flexible job shop scheduling problem has a certain research basis and advantages. Summary of the invention
[0005] The purpose of the present invention is to solve the problems raised in the above-mentioned background technology, and proposes a distributed flexible job shop scheduling optimization system driven by reinforcement learning, which takes minimizing the maximum completion time and minimizing the total energy consumption as the optimization goals to solve the distributed flexible job shop scheduling problem under energy efficiency constraints.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] A distributed flexible job shop scheduling optimization system driven by reinforcement learning, characterized in that the system includes a scheduling sequence initialization module, a hyper-heuristic framework module based on Q-learning, an energy-saving strategy module based on problem characteristics, and a heuristic-based movement acceptance criterion module;
[0008] In the scheduling sequence initialization module, according to the encoding scheme of the three-dimensional vector FA-MA-OS composed of the plant sequence FA, the machine sequence MA and the operation sequence OS, the initial population is constructed using a multi-dimensional heuristic initialization method of three heuristics H1, H2, H3 and a random perturbation R1;
[0009] In the hyper-heuristic framework module based on Q-learning, problem knowledge is extracted according to the characteristics of the research problem, and a neighborhood search operator is designed to improve the efficiency of local search. There are eight different neighborhood search operators, and each neighborhood structure searches a different solution space. Different neighborhood structures can find better solutions. In addition, a self-learning selection strategy is designed in combination with the Q-learning algorithm. The Q-learning algorithm selects the most appropriate low-level heuristic at each step in the population update process. Among them, the eight different neighborhood search operators are regarded as action sets.
[0010] In the energy-saving strategy module based on problem characteristics, energy-saving operations are performed on the current new solution to reduce the energy consumption target value of the solution sequence and accelerate the convergence of the algorithm;
[0011] In the heuristic-based mobile acceptance criterion module, by comparing the fitness values of the new solution with the current solution, the new solution is accepted with probability P. The probability P is determined by the difference in fitness values between the new solution and the current solution. That is, the better the new solution, the greater the acceptance probability, and the worse the new solution, the smaller the acceptance probability.
[0012] Preferably, in the scheduling sequence initialization module, first, the factory sequence FA vector is constructed by H1, and the factory with the lowest load is selected for the workpiece according to the shortest factory load (average processing time in the factory); secondly, the machine sequence MA vector is constructed by H2, and like H1, H2 selects the machine with the lowest load for the workpiece and processes it. If the loads of multiple machines are the same, a machine is randomly selected for processing; then, H3 constructs the operation sequence OS vector according to the remaining processing time of the workpiece, and the workpiece with the longest remaining processing time is processed first; finally, R1 adds random perturbations to the constructed FA-MA-OS sequence to increase the diversity of the population, so that the system can obtain higher-precision results when solving the distributed flexible job shop scheduling problem.
[0013] Preferably, in the hyper-heuristic framework module based on Q-learning, in order to improve the performance of the hyper-heuristic, Q-learning is used as a high-level strategy, and the agent is trained through the feedback information of each iteration. The agent selects the action (low-level heuristic) that best suits the current environment according to the environment (population state) to explore and develop the solution space; the training process adopts an improved ∈-greedy strategy, and its action selection probability is as follows:
[0014]
[0015] Q(s t ,a t )←Q(s t ,a t )+α(r t+1 +γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )) (3)
[0016] Among them, ∈ refers to the sample value drawn from the standard normal distribution at maxFE, maxFE is the maximum number of function evaluations, which is the stopping criterion, Set(A) is the action set of Q-learning, and a * Indicates state s t The action with the largest Q value at the time; From formula (1), it can be seen that as the number of evaluations increases, the value of ∈ will gradually decrease to 0; At the beginning of training, the probability of the agent exploring new actions is almost 40%. As time goes by, the probability of rand < ∈ gradually decreases. According to formula (2), the agent tends to choose the action with the largest Q value. In other words, in the early stages of action selection (i.e., selecting LLH), the agent maintains a certain degree of exploration, but as training proceeds, it is more inclined to use the learned knowledge to guide action selection; The agent's learning experience based on the environment is stored in a Q table, where each row and column in the Q table represents a state and an action, respectively, and each Q value in the Q table is calculated by the Q function, as shown in formula (3);
[0017] Among them, Q(s t ,a t ) means in state s t Take action a t The Q value at the time of training, the parameter α controls the learning weight of each round of training, the range is [0,1], called the learning rate; as shown in formula (1), the value of α gradually decreases from 1. When the value of α decreases from 1 to 0, it means that the agent pays more attention to the reward obtained by the current action. The discount factor is represented by γ, the range is [0-1], max Q(s t+1 ,a t+1 ) refers to t+1 In the state, take a t+1 The maximum Q value of the Q table during action;
[0018] The action set consists of eight neighborhood search operators, and the state is divided according to the normalized value of the population objective function, dividing the state set into 10 non-overlapping parts. For example, when S t ∈[0,0.1], S t =S1; when S t ∈[0.9,1.0], S t =S 10 .
[0019] Specifically, by inserting and exchanging individuals in the population, a neighborhood solution is generated, and a better solution is searched in the current neighborhood solution, that is, a neighborhood solution that minimizes the objective function value is found. The eight neighborhood search operators are introduced as follows:
[0020] N1(LLH1): Randomly select two operations within the critical block and swap their positions;
[0021] N2(LLH2): Randomly select two operations in the critical block and insert the latter operation before the former operation;
[0022] N3(LLH3): randomly select a key operation in the key block and insert it into another machine;
[0023] N4(LLH4): swap head and tail operations within a key block;
[0024] N5(LLH5): Select one operation in a key block and insert it before the head node, and another operation after the tail operation;
[0025] N6-variant (LLH6): Based on the N6 operation, the tail operation in the key block is inserted before the head operation;
[0026] N7 (LLH7): Before the N6 operation, insert the head and tail operations in the key block into any other position in the key block;
[0027] N8 (LLH8): After N7 operation, insert the head and tail operations in the key block to any other position outside the key block.
[0028] Preferably, in the energy-saving strategy module based on problem characteristics, when operation O i,j When arriving at the machine, scan the previous idle time block on the same machine to determine whether there is an operation with a processing time less than the idle time block, and then move operation O i,j Move to this free time block and perform other operations O i,j Move forward; then, execute the above steps to convert the scheduling scheme from semi-active scheduling to active scheduling, reversely traverse the idle time blocks of the machine where the operation is located, and move the operations whose processing time is less than the idle time block to the right; finally, repeat the above steps until the scheduling scheme is converted to full-active scheduling, that is, there is no idle time for operation.
[0029] Preferably, in the heuristic-based move acceptance criterion module, a simulated annealing heuristic method is used as the move acceptance criterion, with probability P maAccept the relatively poor quality solutions in the new group, so as to get rid of the trouble of falling into the local optimum; if the new solution π is improved than the current solution, it will replace the current solution; otherwise, the probability of the new solution being accepted is P ma , the probability is calculated by formula (6):
[0030]
[0031] In formula (6), the difference between the average fitness value of the new solution and the current solution is used to represent the improvement degree of the new solution. From formula (6), it can be concluded that the lower the improvement degree of the new solution (that is, the worse the quality of the new solution), the lower the acceptance probability.
[0032] The beneficial effects of the present invention are:
[0033] 1. The Q-learning algorithm is used to select the high-level strategy of the hyper-heuristic and dynamically select the low-level heuristics. Based on the search state and the historical feedback information of each low-level heuristic, the appropriate neighborhood search operator is selected to further balance the algorithm's exploration and development capabilities of the solution space.
[0034] 2. The present invention designs multiple neighborhood search operators according to the problem characteristics and uses them as low-level heuristics in the selection hyper-heuristic to improve the efficiency of local search.
[0035] 3. Energy-saving operations based on problem knowledge effectively reduce energy consumption and improve solution quality
[0036] 4. The heuristic moving acceptance criterion probabilistically accepts the worse solution, which prevents the algorithm from falling into the local optimum and improves the global search efficiency of the algorithm.
[0037] 5. The logic of the present invention is simple, easy to implement and easy to expand, and the optimizer can be expanded to meet most scheduling problems in the current intelligent manufacturing production field. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 It is a Gantt chart of the distributed flexible job shop scheduling problem in the first embodiment of the invention;
[0039] Figure 2 This is a schematic diagram of the encoding and decoding solution in Embodiment 1 of the invention;
[0040] Figure 3 This is a diagram of the hyper-heuristic framework for selecting in the first embodiment of the invention;
[0041] Figure 4 This is a diagram of the algorithm framework in the first embodiment of the invention;
[0042] Figure 5 This is a schematic diagram of energy-saving operation in Embodiment 1 of the invention;
[0043] Figure 6 Schematic diagram of eight neighborhood search operators in Embodiment 1 of the invention. DETAILED DESCRIPTION
[0044] Embodiment 1
[0045] Schematic diagram of distributed flexible job shop scheduling Figure 1 As shown. The system can optimize the operating efficiency and performance of the distributed flexible job shop scheduling system. The present invention is a solver for solving the distributed flexible job shop scheduling problem, including the following contents:
[0046] Step 1: Design a multi-dimensional heuristic initial population generation strategy;
[0047] Step 2: A hyper-heuristic framework based on reinforcement learning mechanism is designed to adaptively select search operators according to the search state and operator feedback information. Compared with the traditional selection of a single search operator, the search operator is adaptively selected according to the real-time state of the population, and the solution space is explored and developed to varying degrees, which improves the search efficiency of the algorithm;
[0048] Step 3: The neighborhood search operator and energy-saving strategy based on problem characteristics improve the local search efficiency of the algorithm;
[0049] Step 4: Based on the heuristic mobile reception criterion, the algorithm is prevented from falling into the local optimum and the diversity of understanding is ensured.
[0050] Preferably, in step 1, first, according to the three-dimensional vector encoding method, three heuristics (H1, H2, H3) are used to generate the factory sequence (FA), machine sequence (MA) and operation sequence (OS) respectively. The encoding method schematic diagram is as follows: Figure 2 As shown. Secondly, random perturbation R1 is added to the constructed FA-MA-OS sequence to increase the diversity of the population.
[0051] Preferably, in step 2, a hyper-heuristic framework for selecting a reinforcement learning mechanism is designed. The hyper-heuristic framework is selected as Figure 3 As shown in Figure 2, the combined model of SHH and reinforcement learning is as follows: Figure 4 As shown in Figure 2, by training the agent, the agent adaptively selects the search operator to explore and develop the solution space according to the real-time status of the population.
[0052] Preferably, in step 3, eight neighborhood search operators for key operations are designed based on the problem characteristics with insertion, exchange and deletion as basic operations. Figure 5 At the same time, the designed energy-saving operation can effectively reduce energy consumption and improve the quality of the solution. Figure 6 shown.
[0053] Preferably, in step 4, in order to improve the diversity of solutions and avoid the algorithm from falling into a local optimum, a heuristic-based mobile reception criterion is proposed, which accepts new solutions (including inferior solutions) with probability based on the quality of the solutions.
[0054] Embodiment 2
[0055] 1. Population initialization module
[0056] According to the encoding scheme of the three-dimensional vector (FA-MA-OS) composed of factory allocation (FA), machine allocation (MA) and operation sequence (OS), the initial population is constructed using a multi-dimensional heuristic initialization method with three heuristics (H1, H2, H3) and a random perturbation (R1). Among them, H1, H2, H3 and R1 are: selecting the factory with the lowest load for the workpiece according to the shortest average processing time in the factory (heuristic method H1), selecting the machine with the lowest load for the workpiece and processing it, if multiple machines have the same load, randomly selecting a machine for processing (heuristic method H2), the workpiece with the longest remaining processing time is processed first (heuristic method H3) and randomly exchanging the sequence numbers of two positions in the FA-MA-OS sequence to form a new individual (random perturbation R). First, the FA vector is constructed by H1, that is, according to the factory load (the shortest average processing time in the factory), the factory with the lowest load is selected for the workpiece. Secondly, the MA vector is constructed by H2. Same as H1, H2 selects the machine with the lowest load for the workpiece and processes it, if multiple machines have the same load, randomly selects a machine for processing. H3 constructs the OS vector according to the remaining processing time of the workpiece, and the workpiece with the longest remaining processing time is processed first. Finally, R1 adds random perturbations to the constructed FA-MA-OS sequence to increase the diversity of the population.
[0057] 2. High-level strategy based on Q-learning
[0058] In order to improve the performance of the hyper-heuristic, Q-learning is used as a high-level strategy. The agent is trained through the feedback information of each iteration. The agent selects the action (low-level heuristic) that best suits the current environment according to the environment (population state) to explore and develop the solution space. The training process adopts an improved ∈-greedy strategy, and its action selection probability is as follows:
[0059]
[0060] Q(s t ,a t )←Q(s t ,a t )+α(r t+1 +γmaxQ(a t+1 ,a t+1 )-Q(s t ,at )) (3)
[0061] Here, ∈ refers to the sample value drawn from the standard normal distribution at maxFE. maxFE is the maximum number of function evaluations and the stopping criterion. Set(A) is the action set of Q-learning, a * Indicates state s t The action with the largest Q value at that time. From formulas (1) and (2), it can be seen that as the number of evaluations increases, the value of ∈ will gradually decrease to 0. At the beginning of training, the probability of the agent exploring new actions is almost 40%. As time increases, the probability of rand < ∈ gradually decreases. According to formula (2), the agent tends to choose the action with the largest Q value. In other words, in the early stages of action selection (i.e., selecting LLH), the agent maintains a certain degree of exploration, but as training proceeds, it is more inclined to use the learned knowledge to guide action selection. The agent's learning experience based on the environment is stored in a Q table. Each row and column in the Q table represents a state and an action, respectively. Each Q value in the Q table is calculated by the Q function, as shown in formula (3).
[0062] Among them, Q(s t ,a t ) means in state s t Take action a t The Q value at the time. The parameter α controls the learning weight of each round of training, and the range is [0,1], which is called the learning rate. As shown in formula (1), the value of α gradually decreases from 1. When the value of α decreases from 1 to 0, it means that the agent pays more attention to the reward obtained by the current action. The discount factor is represented by γ, and the range is [0-1]. max Q(s t+1 ,a t+1 ) refers to t+1 In the state, take a t+1 The maximum Q value of the Q table during action.
[0063] In QLSHH, the action set consists of eight neighborhood search operators, and the state is divided according to the normalized value of the population objective function, dividing the state set into 10 non-overlapping parts. For example, when S t ∈[0,0.1], S t =S1; when S t ∈[0.9,1.0], S t =S 10 .
[0064]
[0065] Reward mechanism: By calculating the average crowding distance of individuals in the PF of each generation of the population To compare the diversity of optimal solutions.t and Pop t+1 The reward value can be obtained by evaluating the dominance relationship between individuals in the PF and the diversity and average fitness value of the optimal individuals. The calculation formulas for the and reward values are shown in formulas (4) and (5):
[0066] Among them, t represents the iteration to the tth generation, N PF Represents the number of all PF individuals. λ1 and λ2 are the base reward and base penalty values for each reward evaluation, set to 1 and -1 respectively.
[0067] 3. Neighborhood search operator based on problem knowledge
[0068] Neighborhood search is a common method for solving job shop scheduling problems. Its basic idea is to search for a better solution in the neighborhood of the solution. By inserting and exchanging individuals in the population, a neighborhood solution is generated, and a better solution is searched in the current neighborhood solution, that is, the neighborhood solution that minimizes the objective function value is found. The eight neighborhood search operators are introduced in the following table:
[0069]
[0070]
[0071] 4. Energy-saving strategies
[0072] Scheduling can be divided into active scheduling (AS), semi-active scheduling (SAS) and full-active scheduling (FAS). Based on problem knowledge, the idle time can be effectively reduced by converting semi-active scheduling to active scheduling and then to full-active scheduling. The specific steps of the energy-saving strategy based on problem characteristics are as follows:
[0073] (1) When an OS operation arrives, scan the previous idle time block on the same machine to determine whether there is an operation with a processing time less than the idle time block, and then replace the operation O i,j Move to this free time block and perform other operations O i,j Move forward.
[0074] (2) After executing step 1 to convert the scheduling scheme from SAS to AS, reversely traverse the idle time blocks of the machine where the operation is located, and move the operations whose processing time is less than the idle time block to the right.
[0075] (3) Repeat the above steps until the scheduling scheme is converted to FAS.
[0076] The schematic diagram of the energy saving strategy is as follows Figure 6 shown.
[0077] 5. Mobile Acceptance Criteria
[0078] Whether to accept the newly generated solution is determined by the move acceptance mechanism in the hyperheuristic. In QLSHH, the simulated annealing heuristic method is used as the move acceptance criterion, with probability P ma Accept the relatively poor quality solutions in the new population, so as to get rid of the trouble of being trapped in the local optimum. If the new solution π is improved over the current solution, it replaces the current solution. Otherwise, the probability of the new solution being accepted is P ma , the probability is calculated by formula (6):
[0079]
[0080] In formula (6), the difference between the average fitness value of the new solution and the current solution is used to represent the improvement degree of the new solution. From formula (6), it can be concluded that the lower the improvement degree of the new solution (that is, the worse the quality of the new solution), the lower the acceptance probability.
[0081] Embodiment 3
[0082] System Process
[0083] A distributed flexible job shop scheduling optimization system driven by reinforcement learning is mainly composed of five parts: a mixed population initialization strategy to generate an initial population, a global search strategy based on a reinforcement learning mechanism, a neighborhood search operator based on problem knowledge, an energy-saving strategy based on problem characteristics, and a mobile acceptance criterion based on heuristics. In the initialization stage, a high-quality initial population is generated by combining construction heuristics and randomization. In terms of global search, a high-level selection strategy based on Q-learning is designed. The intelligent agent dynamically selects appropriate low-level heuristics according to the state of the population at different optimization stages and the feedback information after each iteration. In terms of local search, the neighborhood search operator based on problem knowledge can fully explore and develop a large and complex problem solution space. The heuristic-based mobile acceptance criterion prevents the QLSHH algorithm from falling into local optimality. The QLSHH pseudocode is shown in Algorithm 1, and the algorithm framework diagram is shown in Figure 4 shown.
[0084]
[0085] Finally, through simulation experiments, the experimental results show that the solver proposed in this invention is superior to other solvers in terms of solution time and solution quality.
[0086] Finally, it should be noted that: Obviously, the above embodiments are only examples for clearly explaining the present invention, and are not intended to limit the implementation methods. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from this are still within the scope of protection of the present invention.
Claims
1. A distributed flexible job shop scheduling optimization system driven by reinforcement learning, characterized in that: The system includes a scheduling sequence initialization module, a hyper-heuristic framework module based on Q-learning, an energy-saving strategy module based on problem characteristics, and a move acceptance criterion module based on heuristics; In the scheduling sequence initialization module, according to the encoding scheme of the three-dimensional vector FA-MA-OS composed of the plant sequence FA, the machine sequence MA and the operation sequence OS, the initial population is constructed using a multi-dimensional heuristic initialization method of three heuristics H1, H2, H3 and a random perturbation R1; In the hyper-heuristic framework module based on Q-learning, problem knowledge is extracted according to the characteristics of the research problem, and a neighborhood search operator is designed to improve the efficiency of local search. There are eight different neighborhood search operators, and each neighborhood structure searches a different solution space. Different neighborhood structures can find better solutions. In addition, a self-learning selection strategy is designed in combination with the Q-learning algorithm. The Q-learning algorithm selects the most appropriate low-level heuristic at each step in the population update process. Among them, the eight different neighborhood search operators are regarded as action sets. In the energy-saving strategy module based on problem characteristics, energy-saving operations are performed on the current new solution to reduce the energy consumption target value of the solution sequence and accelerate the convergence of the algorithm; In the heuristic-based mobile acceptance criterion module, by comparing the fitness values of the new solution with the current solution, the new solution is accepted with probability P. The probability P is determined by the difference in fitness values between the new solution and the current solution. That is, the better the new solution, the greater the acceptance probability, and the worse the new solution, the smaller the acceptance probability.
2. The distributed flexible job shop scheduling optimization system driven by reinforcement learning according to claim 1 is characterized in that: In the scheduling sequence initialization module, first, the factory sequence FA vector is constructed by H1, and the factory with the lowest load is selected for the workpiece according to the shortest factory load; secondly, the machine sequence MA vector is constructed by H2. Like H1, H2 selects the machine with the lowest load for the workpiece and processes it. If the loads of multiple machines are the same, a machine is randomly selected for processing; then, H3 constructs the operation sequence OS vector according to the remaining processing time of the workpiece, and the workpiece with the longest remaining processing time is processed first; finally, R1 adds random perturbations to the constructed FA-MA-OS sequence to increase the diversity of the population, so that the system can obtain more accurate results when solving the distributed flexible job shop scheduling problem.
3. The distributed flexible job shop scheduling optimization system driven by reinforcement learning according to claim 2 is characterized by: In the hyper-heuristic framework module based on Q-learning, in order to improve the performance of hyper-heuristic, Q-learning is used as a high-level strategy. The agent is trained through the feedback information of each iteration. The agent selects the action (low-level heuristic) that best suits the current environment according to the environment (population state) to explore and develop the solution space; the training process adopts an improved ∈-greedy strategy, and its action selection probability is as follows: Q(s t ,a t )←Q(s t ,a t )+α(r t+1 +γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )) (3) Among them, ∈ refers to the sample value drawn from the standard normal distribution at maxFE, maxFE is the maximum number of function evaluations, which is the stopping criterion, Set(A) is the action set of Q-learning, and a * Indicates state s t The action with the largest Q value when the agent is selected; from formula (1), it can be seen that as the number of evaluations increases, the value of ∈ will gradually decrease to 0; at the beginning of training, the probability of the agent exploring new actions is almost 40%. As time goes by, the probability of rand>1-∈ holding the condition gradually decreases, and the agent tends to choose the action with the largest Q value; in other words, in the early stage of action selection (i.e., selecting LLH), the agent maintains a certain degree of exploration, but as training proceeds, it is more inclined to use the learned knowledge to guide action selection; the agent's learning experience based on the environment is stored in a Q table, where each row and column in the Q table represents a state and an action, respectively, and each Q value in the Q table is calculated by the Q function, as shown in formula (3); Among them, Q(s t ,a t ) means in state s t Take action a t The Q value at the time of training, the parameter α controls the learning weight of each round of training, the range is [0,1], called the learning rate; as shown in formula (1), the value of α gradually decreases from 1. When the value of α decreases from 1 to 0, it means that the agent pays more attention to the reward obtained by the current action. The discount factor is represented by γ, the range is [0-1], maxQ(s t+1 ,a t+1 ) refers to t+1 In the state, take a t+1 The maximum Q value of the Q table during action; The action set consists of eight neighborhood search operators, and the state is divided according to the normalized value of the population objective function, dividing the state set into 10 non-overlapping parts. For example, when S t ∈[0,0.1], S t =S1; when S t ∈[0.9,1.0], S t =S 10 .
4. The distributed flexible job shop scheduling optimization system driven by reinforcement learning according to claim 3 is characterized by: When operating i,j When arriving at a machine, scan the previous idle time block on the same machine to determine whether there is an operation with a processing time less than the idle time block (where O i,j is the jth operation of the ith workpiece). i,j Move to this free time block and perform other operations O i,j Move forward; then, after executing the above steps to convert the scheduling scheme from semi-active scheduling to active scheduling, reversely traverse the idle time blocks of the machine where the operation is located, and move the operations whose processing time is less than the idle time block to the right; Finally, repeat the above steps until the scheduling scheme is converted to full-active scheduling, that is, there is no operational idle time.
5. The distributed flexible job shop scheduling optimization system driven by reinforcement learning according to claim 4 is characterized by: By inserting and exchanging individuals in the population, a neighborhood solution is generated, and a better solution is searched in the current neighborhood solution, that is, the neighborhood solution that minimizes the objective function value is found. The eight neighborhood search operators are introduced as follows: N1(LLH1): Randomly select two operations within the critical block and swap their positions; N2(LLH2): Randomly select two operations in the critical block and insert the latter operation before the former operation; N3(LLH3): randomly select a key operation in the key block and insert it into another machine; N4(LLH4): swap head and tail operations within a key block; N5(LLH5): Select one operation in a key block and insert it before the head node, and another operation after the tail operation; N6-variant (LLH6): Based on the N6 operation, the tail operation in the key block is inserted before the head operation; N7 (LLH7): Before the N6 operation, insert the head and tail operations in the key block into any other position in the key block; N8 (LLH8): After N7 operation, insert the head and tail operations in the key block to any other position outside the key block.
6. The distributed flexible job shop scheduling optimization system driven by reinforcement learning according to claim 5 is characterized by: In the heuristic-based move acceptance criterion module, the simulated annealing heuristic method is used as the move acceptance criterion, with probability P ma Accept the relatively poor quality solutions in the new group, so as to get rid of the trouble of falling into the local optimum; if the new solution π is improved than the current solution, it will replace the current solution; otherwise, the probability of the new solution being accepted is P ma , the probability is calculated by formula (6): In formula (6), the difference between the average fitness value of the new solution and the current solution is used to represent the improvement degree of the new solution. From formula (6), it can be concluded that the lower the improvement degree of the new solution, that is, the worse the quality of the new solution, the lower the acceptance probability.
Citation Information
Cited By
Power grid metering material verification and distribution collaborative optimization method and device and storage medium
CN121052760A
Flexible workshop scheduling optimization method based on meta-heuristic algorithm
CN121684562A
