A hybrid flow shop rescheduling method and system based on Q-learning

Abstract

The application relates to the technical field of intelligent manufacturing, in particular to a hybrid flow shop rescheduling method and system based on Q-learning. A multi-objective optimization model is established, with total energy consumption cost, carbon trading cost and total weighted early and late penalty as optimization objectives; evolution operations are performed on the current population based on adaptive variable neighborhood search driven by Q-learning; for the idle period of a machine tool in the scheduling scheme, a decision is made on whether to perform a shutdown-restart operation or keep the standby state based on an energy balance point; in response to a new workpiece arrival event, partial rescheduling is performed, the scheduling state of the completed and processing workpieces is inherited, and the unstarted workpiece and the newly arrived workpiece are combined for local rescheduling. The purpose is to adaptively adjust the neighborhood search strategy according to the real-time search state to break through the search efficiency bottleneck of fixed operator selection, and simultaneously realize rapid response to the dynamic arrival of new workpieces and effective control of the energy consumption of the idle period of the machine tool.

Description

Technical Field

[0001] This application relates to the field of intelligent manufacturing technology, and in particular to a hybrid flow shop rescheduling method and system based on Q-learning. Background Technology

[0002] Modern manufacturing system scheduling technology is rapidly evolving from traditional static single-objective optimization to dynamic, multi-objective, and green intelligent manufacturing. Existing technologies typically employ non-dominated sorting genetic algorithms (NSGA-II / NSGA-III) as a multi-objective optimization framework to solve production scheduling problems in hybrid flow shops. This approach establishes a model with multiple optimization objectives, including total energy cost, carbon trading cost, and weighted advance / delay penalties. It encodes and evolves the workpiece processing sequence using fixed crossover, mutation, and neighborhood search operators, achieving a relatively optimal Pareto solution set under static conditions.

[0003] While this solution can achieve some success in static scheduling scenarios, in actual production, hybrid assembly line workshops face the coupled influence of multiple complex factors such as sequence-related setup time, random arrival of new workpieces, and dynamic changes in machine tool status. Existing technologies are often limited by the following factors, leading to a significant decrease in scheduling performance: First, the selection probabilities of various operators in the NSGA-II framework are fixed or determined by human experience before the search begins, making it impossible to dynamically adjust according to the real-time status of the search process. This makes the algorithm prone to getting trapped in local optima in complex high-dimensional multi-objective solution spaces, and the search efficiency drops sharply as the problem size increases. Second, when new workpieces arrive dynamically during the production process, the existing full rescheduling strategy requires recalculating the scheduling scheme for all workpieces, resulting in high computational overhead and slow response speed. While the right-shift strategy or idle time window insertion strategy has a lower computational cost, it cannot guarantee the global optimality of the scheduling scheme. Third, existing models generally do not fully consider the energy consumption contribution corresponding to the inter-process setup time, and lack an energy-saving mechanism that dynamically decides whether to shut down or keep the machine in standby mode based on the actual idle state of the machine tool. This causes the machine tool to run continuously during idle periods, resulting in unnecessary energy consumption and carbon emissions.

[0004] The underlying reasons are as follows: First, the fixed-probability operator selection mechanism cannot perceive the structural characteristics changes of the solution space at different search stages: strong perturbations are needed in the early stages of the search to promote global exploration, while fine-grained local searches are needed in the later stages to accelerate convergence. Fixed strategies cannot make differentiated responses according to the different needs of the search stages. Second, the dynamic insertion of new workpieces essentially breaks the feasibility of the original scheduling scheme. The reason why full rescheduling is inefficient is that it ignores the fact that the scheduling states of completed and currently being processed workpieces can be inherited and reused. Each dynamic event requires re-optimization from scratch, resulting in a large amount of redundant computation. Third, the optimization of machine tool energy consumption is not only a numerical optimization problem, but also involves the trade-off between the energy consumption cost of switching machine tools between different physical operating states and the actual energy-saving benefits. Existing schemes lack an effective method to systematically embed this physical-level trade-off into the optimization decision-making process.

[0005] Therefore, how to enable multi-objective evolutionary algorithms to adaptively adjust neighborhood search strategies based on real-time search status in mixed flow workshop scheduling scenarios that include sequence-related setting time and dynamic arrival of new workpieces in order to overcome the search efficiency bottleneck of fixed operator selection, while achieving rapid response to the dynamic arrival of new workpieces and effective control of energy consumption during machine tool idle periods, has become an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0006] The main objective of this application is to provide a hybrid flow shop rescheduling method based on Q-learning, which aims to adaptively adjust the neighborhood search strategy according to the real-time search status to overcome the search efficiency bottleneck of fixed operator selection, while realizing rapid response to the dynamic arrival of new workpieces and effective control of energy consumption during machine tool idle periods.

[0007] To achieve the above objectives, this application provides a method for readjusting a mixed flow workshop, the method comprising: S10, Establish a multi-objective optimization model, wherein the multi-objective optimization model takes total energy consumption cost, carbon trading cost and total weighted advance and delay penalties as optimization objectives, and sequence-related setting time, machine tool speed level, processing sequence and energy consumption as constraints. S20, an initial population is generated using a hybrid initialization strategy. The hybrid initialization strategy consists of a heuristic solution of a first proportion and a random solution of a second proportion, and includes a sequential crossover operator, a cyclic crossover operator, and a sequence-based crossover operator. S30, based on Q-learning-driven adaptive variable neighborhood search, perform iterative updates on the initial population; S40, calculate the adaptive crowding distance of each initial population, and use the adaptive crowding distance as a screening index to select the next generation population through fast non-dominated sorting and elite retention strategy; After obtaining the current generation's scheduling scheme, S50 determines whether the target machine tool in the idle period should perform a shutdown-restart operation based on the energy consumption balance point; S60, in response to a newly arrived workpiece detected during production, identifies and locks completed and / or in-process workpieces and inherits their scheduling state, and merges the unstarted workpieces with the newly arrived workpieces for local rescheduling.

[0008] Optionally, the objective function of the multi-objective optimization model described in S10 includes: (1) In the formula, Indicates the total processing energy consumption of all machine tools. Indicates the total set energy consumption of all machine tools. This represents the total idle energy consumption of all machine tools. This indicates the total restart energy consumption of all machine tools; Indicates time period Electricity price; Indicates total energy consumption; This indicates the number of electricity price periods within a day; This indicates the total cost of carbon trading; Indicates the carbon emission coefficient per unit of energy consumption; (2) In the formula, Indicates carbon emission quotas; Indicates the carbon trading price; (3) In the formula, This indicates the total weighted average penalties for early and late payments; , Indicates workpiece Delayed completion penalty coefficient, workpiece Early completion penalty coefficient; , Indicates the earliest delivery date and the latest delivery date of the workpiece; This represents the completion time of workpiece j in the last stage. There are a total of g stages.

[0009] Optionally, the constraints of the multi-objective optimization model described in S10 include: (1) Processing timing constraints: (4) In the formula, Indicates the maximum completion time; N represents the set of workpieces; (5) In the formula, For binary decision variables, if in stage workpiece Assigned to machine tools If the process is to be completed, then the value is 1; otherwise, it is 0. M represents the stage set. (6) (7) In the formula, For binary decision variables, if in stage machine tools Above, workpiece In the workpiece If it is a pre-processing step, take 1; otherwise, take 0. Indicates if in the stage machine tools If workpiece k is processed before workpiece j, then the value is 1; otherwise, the value is 0. Indicates if in the stage Workpiece k is assigned to the machine tool If the machining process is performed, then the value is 1; otherwise, it is 0. L represents the set of machine tools. (8) In the formula, Indicates the number of machine tool speed classes; Represents a set of machine tool speed levels; Indicates if in the stage workpiece At speed level If it is being processed, take 1; otherwise, take 0. (9) In the formula, Indicates the stage workpiece In machine tool The actual processing time; Indicates workpiece In the stage Standard processing time; Indicates the machine tool speed level The processing speed is reduced; (2) Time constraints: (10) In the formula, Indicates workpiece In the stage Start processing time; Indicates workpiece In the stage Completion time; (11) In the formula, Indicates workpiece The start processing time of stage i+1; (12) In the formula, H represents a positive number; (13) In the formula, Indicates the stage machine tool Start time of processing; Indicates that workpiece l is in stage The completion time.

[0010] (14) In the formula, Representation phase The setting time for machine tool l; Indicates the stage workpiece In the workpiece The setting time for the machining tool during subsequent processing; (15) (16) In the formula, This represents the total machine tool processing time across all stages; (17) In the formula, Representation phase Total free time; Indicates the stage machine tool The time for stopping processing; Indicates the stage machine tool Start time of processing; (18) In the formula, This represents the total idle time of the machine tool across all stages; (19) In the formula, This indicates the total machine tool restart time across all stages; (20) (3) Energy consumption constraints (twenty one) In the formula, Indicates the machine tool speed level Energy consumption per unit time of machine tool operation; (twenty two) (twenty three) In the formula, Indicates the stage Unit idle energy consumption; (twenty four) In the formula, Indicates the stage Unit restart energy consumption; (25).

[0011] Optionally, the multi-objective optimization model further includes a stability index, the calculation expression of which is:

[0012] in, .

[0013] Optionally, S30 includes: S31, Based on the first Q table, determine the target damage intensity corresponding to the current first state; S32, perform the destruction-reconstruction operation corresponding to the target destruction intensity, remove and re-insert the corresponding number of workpieces from the current solution's workpiece sequence; S33, Based on the second Q table, determine the target local search operator type corresponding to the current second state; S34, Execute the selected local search operator to perform local optimization on the current solution; S35, calculate the normalized reward value of the multi-objective improvement, and update the first Q table and / or the second Q table, wherein the formula for calculating the normalized reward value is:

[0014] in, This represents the initial solution. This represents the improved solution; The initial solution is represented in the th case. The target value on each target, The improved solution is represented in the th... The target value on each target; Indicates the number of the current population The minimum value of each objective. Indicates the number of the current population The maximum value of each target.

[0015] Optionally, the S50 includes: When the predicted idle time of the target machine tool is greater than the energy consumption balance point, the target machine tool is controlled to perform a shutdown-restart operation; When the predicted idle time is less than or equal to the energy consumption balance point, the target machine tool is controlled to remain in standby mode.

[0016] Optionally, the formula for calculating the energy-saving balance point in S50 is:

[0017] In the formula, It is a machine tool The energy consumption balance point is defined as the critical value of idle time when the energy cost of restarting is equal to the energy cost of keeping idle. It is the basic equilibrium point; Indicates workpiece In the stage By speed level Processed; It is an adjustment coefficient that adjusts the energy consumption balance point according to the processing speed level.

[0018] Optionally, the adaptive congestion distance described in S40 is calculated using the following expression:

[0019] In the formula, Indicates the first The total crowding distance of each solution; This represents the total number of objective functions; Indicates the index of the current solution; , They respectively represent the first After sorting the targets, compare them with the current solution. The immediately preceding and following solutions; Indicates the first The solution is at the th solution. Function values ​​on each objective function; This indicates that in the current population, the [number]th ... The maximum value of each objective function; This indicates that in the current population, the [number]th ... The minimum value on an objective function.

[0020] In addition, to achieve the above objectives, this application also provides a hybrid flow shop rescheduling system, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the Q-learning-based hybrid flow shop rescheduling method as described in any of the preceding claims.

[0021] This application has at least the following beneficial effects: (1) Considering intelligent manufacturing and breaking through the traditional limitation of only focusing on the construction period, a multi-objective model was established to simultaneously optimize total energy consumption cost, carbon trading cost and weighted early / late penalty, taking into account both economic efficiency and environmental protection.

[0022] (2) A hybrid initialization method combining multi-objective heuristic rules and random strategies is adopted, and an adaptive crowding distance mechanism is introduced to ensure the quality of the initial population and the uniformity of the Pareto solution set.

[0023] (3) The Q-learning method is used to replace the traditional fixed probability selection. The most effective neighborhood search operator is selected adaptively according to the real-time search state, which solves the problem of operator selection rigidity and easy getting trapped in local optima.

[0024] (4) Based on the energy balance point calculation, intelligent decision-making is made on whether idle machine tools should remain in standby or be shut down and restarted, thereby significantly reducing standby energy consumption and carbon emissions without changing the completion time.

[0025] (5) A hybrid decoding mechanism based on list scheduling and earliest completion time rule is proposed for dynamic scheduling. Combined with a partial rescheduling strategy, it effectively handles the problem of dynamic insertion of new workpieces and introduces a stability index to balance system fluctuations. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the hybrid flow shop rescheduling method based on Q-learning involved in the embodiments of this application; Figure 2 This is an overall framework diagram of the QVNS involved in the embodiments of this application; Figure 3 This is a flowchart illustrating the fast non-dominated sorting method used in embodiments of this application; Figure 4 This is a schematic diagram illustrating the encoding involved in the embodiments of this application; Figure 5 This is a box plot of HV involved in the embodiments of this application; Figure 6 Box plot of IGD values ​​involved in the embodiments of this application; The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0027] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0028] Example 1 Reference Figure 1 This embodiment provides a hybrid flow shop rescheduling method based on Q-learning, the method including the following steps: S10, Establish a multi-objective optimization model, wherein the multi-objective optimization model takes total energy consumption cost, carbon trading cost and total weighted advance and delay penalties as optimization objectives, and sequence-related setting time, machine tool speed level, processing sequence and energy consumption as constraints. In this embodiment, a formal mathematical model is first performed on the dynamic hybrid flow shop rescheduling problem (SDST-D-HFSP) with sequence-dependent set times and new workpiece arrivals. The input to this problem includes: a production system containing One workpiece, Each processing stage and A machine tool. The workpiece index set is denoted as... The stage index set is denoted as ,in Representation phase The number of parallel machine tools on the platform, each parallel machine tool consisting of... Index. Each workpiece Include Step 1: This corresponds to the processing requirements at each stage. The initial set of workpieces to be processed is... The newly arrived set of workpieces is Each workpiece must be processed on one machine tool at each stage, with each stage corresponding to one operation. When a new workpiece arrives, an event-triggered partial rescheduling strategy is employed. At this point, the workpiece is categorized as completed, in-process, or not yet started.

[0029] Further, and optionally, the objective function of the multi-objective optimization model includes: (1) In the formula, Indicates the total processing energy consumption of all machine tools. Indicates the total set energy consumption of all machine tools. This represents the total idle energy consumption of all machine tools. This indicates the total restart energy consumption of all machine tools; Indicates time period Electricity price; Indicates total energy consumption; This indicates the number of electricity price periods within a day; This indicates the total cost of carbon trading; Indicates the carbon emission coefficient per unit of energy consumption; (2) In the formula, Indicates carbon emission quotas; Indicates the carbon trading price; (3) In the formula, This indicates the total weighted average penalties for early and late payments; , Indicates workpiece Delayed completion penalty coefficient, workpiece Early completion penalty coefficient; , Indicates the earliest delivery date and the latest delivery date of the workpiece; This represents the completion time of workpiece j in the last stage. There are a total of g stages.

[0030] In the above formula, formula (1) represents the total energy cost, which includes processing energy consumption, setup energy consumption, idle energy consumption, and restart energy consumption. Formula (2) represents the total carbon trading cost, where the portion exceeding the carbon emission limit needs to be paid according to the carbon trading price. Formula (3) represents the total weighted penalty for advance and delay for all workpieces. Penalties will be incurred if the final completion time of a workpiece is not within the delivery period. Early completion may increase inventory costs, while delayed completion may cause orders to miss the delivery period. Therefore, minimizing TWET is one of the optimization objectives of this study.

[0031] Furthermore, and optionally, the constraints of the multi-objective optimization model include: (1) Processing timing constraints: (4) In the formula, Indicates the maximum completion time; N represents the set of workpieces; (5) In the formula, For binary decision variables, if in stage workpiece Assigned to machine tools If the process is to be completed, then the value is 1; otherwise, it is 0. M represents the stage set. (6) (7) In the formula, For binary decision variables, if in stage machine tools Above, workpiece In the workpiece If it is a pre-processing step, take 1; otherwise, take 0. Indicates if in the stage machine tools If workpiece k is processed before workpiece j, then the value is 1; otherwise, the value is 0. Indicates if in the stage Workpiece k is assigned to the machine tool If the machining process is performed, then the value is 1; otherwise, it is 0. L represents the set of machine tools. (8) In the formula, Indicates the number of machine tool speed classes; Represents a set of machine tool speed levels; Indicates if in the stage workpiece At speed level If it is being processed, take 1; otherwise, take 0. (9) In the formula, Indicates the stage workpiece In machine tool The actual processing time; Indicates workpiece In the stage Standard processing time; Indicates the machine tool speed level The processing speed is reduced; Formula (4) defines the maximum completion time as the maximum of the completion times of all jobs in the final stage. Formula (5) is an allocation constraint that ensures that each workpiece is assigned to exactly one machine tool in each stage. Formulas (6)-(7) enforce the disjunction priority relationship between any two workpieces processed on the same machine tool, thereby preventing time overlap. Formula (8) is a speed constraint that ensures that each workpiece assigned to the machine tool is assigned to exactly one speed level. Formula (9) is the machining time calculation, which calculates the actual machining time based on the standard machining time and the selected speed level.

[0032] (2) Time constraints: (10) In the formula, Indicates workpiece In the stage Start processing time; Indicates workpiece In the stage Completion time; (11) In the formula, Indicates workpiece The start processing time of stage i+1; (12) In the formula, H represents a positive number; (13) In the formula, Indicates the stage machine tool Start time of processing; Indicates that workpiece l is in stage The completion time.

[0033] (14) In the formula, Representation phase The setting time for machine tool l; Indicates the stage workpiece In the workpiece The setting time for the machining tool during subsequent processing; (15) (16) In the formula, This represents the total machine tool processing time across all stages; (17) In the formula, Representation phase Total free time; Indicates the stage machine tool The time for stopping processing; Indicates the stage machine tool Start time of processing; (18) In the formula, This represents the total idle time of the machine tool across all stages; (19) In the formula, This indicates the total machine tool restart time across all stages; (20) Formula (10) describes the relationship between the start time, actual processing time, and finish time of each workpiece. Formula (11) reflects the sequential constraints of the flow shop, requiring that a workpiece can only begin processing in the second stage after the first stage is completed. Formula (12) considers the sequence-dependent setup times between adjacent workpieces on the same machine tool. Formula (13) ensures that each workpiece meets the earliest available time requirement of its assigned machine. Formulas (14)–(15) summarize the sequence-dependent setup times of all stages and machine tools. Formula (16) sums the actual processing times of all workpieces. Formulas (17)–(19) identify idle periods and distinguish between standby mode and shutdown / restart mode based on a threshold for idle duration. Formula (20) imposes non-negativity constraints on all time-related decision variables.

[0034] (3) Energy consumption constraints (twenty one) In the formula, Indicates the machine tool speed level Energy consumption per unit time of machine tool operation; (twenty two) (twenty three) In the formula, Indicates the stage Unit idle energy consumption; (twenty four) In the formula, Indicates the stage Unit restart energy consumption; (25).

[0035] Formulas (21)–(25) quantify energy consumption during processing, setting, idle, standby and restart states.

[0036] For ease of reference, the meanings of the above parameters are summarized in Tables 1 to 3: Table 1. Key Indicators

[0037] Table 2. Key Parameters

[0038] Table 3. Decision Variables

[0039] S20, an initial population is generated using a hybrid initialization strategy. The hybrid initialization strategy consists of a heuristic solution of a first proportion and a random solution of a second proportion, and includes a sequential crossover operator, a cyclic crossover operator, and a sequence-based crossover operator. In this embodiment, the quality and diversity of the initial population have a decisive impact on the search performance of the evolutionary algorithm. If the initial population is generated completely randomly, although diversity can be guaranteed, the quality of individuals is poor, and the algorithm converges slowly; if it is generated entirely by heuristic methods, although the quality of individuals is high, the diversity is insufficient, and it is easy to get trapped in local optima. To take into account the advantages of both, this embodiment adopts a hybrid initialization strategy.

[0040] The heuristic solution component consists of four methods evenly distributed: single-objective NEH generates dedicated batches of solutions for each of the three objectives (TEC, CTC, and TWET); multi-objective NEH optimizes all objectives simultaneously; LPT rules enhanced with random perturbations promote population diversity; and randomly generated solutions ensure broad coverage of the solution space. This design, through NEH-based heuristics, guarantees both population quality and diversity, effectively mitigating the premature convergence problem that may occur during subsequent NSGA-II optimization.

[0041] In the crossover operation, this embodiment designs a hybrid crossover strategy that integrates three classic crossover operators: sequential crossover (OX), cyclic crossover (CX), and sequence-based crossover (OBX). Through a probability-distribution-based roulette wheel selection mechanism, for each pair of parent individuals generated through tournament selection, the system randomly selects one of the crossover operators to perform the operation, thereby generating two offspring individuals. This hybrid strategy effectively balances convergence speed and population diversity: the OX operator focuses on preserving the relative order information between artifacts, the CX operator emphasizes maintaining the absolute position information of artifacts in the sequence, while the OBX operator achieves a balance between exploration and exploitation by inheriting partial parent gene segments.

[0042] The mutation operation employs a dual strategy to balance the breadth and depth of the search. The primary mutation uses a three-part rearrangement mechanism, randomly selecting two split points to divide the sequence into three parts, and then randomly choosing a recombination method for structural reconstruction. This large-scale mutation helps the algorithm escape local optima. The secondary mutation uses local reversal mutation, randomly selecting a continuous subsequence and reversing its internal order, achieving fine-tuning of the optimal structure through small-scale perturbations. Both mutations are executed with preset probabilities and affect independent copies of individuals to maintain population integrity, thus constructing a hierarchical mutation system that collaboratively supports global exploration and local optimization.

[0043] S30, based on Q-learning-driven adaptive variable neighborhood search, perform iterative updates on the initial population; In this embodiment, the Q-learning-driven adaptive variable neighborhood search mechanism models the operator selection of variable neighborhood search (VNS) as a Markov decision process, achieving dynamic control of the search behavior through a two-layer state-action framework. In each generation of the algorithm's evolution, a destruction-reconstruction (DC) operation is first performed on each individual in the population to achieve macroscopic perturbation, followed by a local search to achieve microscopic optimization, and finally, the reward value is calculated based on the multi-objective improvement amount, and the Q-table is updated. The entire search process does not rely on fixed operator selection probabilities but rather adaptively adjusts the search strategy to meet the needs of different search stages by learning feedback signals from historical search experience.

[0044] Figure 2 The overall framework of QVNS is shown. The Q-learning process requires defining a set of states, actions, and a reward function.

[0045] (1) State space and action space State and action together define the information that an agent can perceive and the feasible decisions it can make. This paper uses neighborhood structure and local search to represent the state space, which not only controls the state space within a limited range but also effectively captures the characteristics of the solution. Considering that the perturbation process and the local search process are independent of each other, their respective state sets are represented as S1 and S2.

[0046] State S1: Represents the strength of the destruction-reconstruction operator used, that is, the number of artifacts removed and re-inserted from the current solution. It controls the breadth of the algorithm's search in the solution space.

[0047] State S2: Represents the type of variable neighborhood search operator currently used, which controls the search depth of the algorithm within the local region.

[0048] This Q-learning framework employs a two-layer state-action strategy. In each generation of evolution, the algorithm first selects a destructive action (Action1) from Q-table 1 based on state S1 and performs a macroscopic destruction-reconstruction operation to escape local optima. Subsequently, it selects a neighborhood operator (Action2) from Q-table 2 based on state S2 and performs a microscopic variable neighborhood search. The reward function comprehensively considers the relative improvement in energy consumption, carbon emissions, and delivery penalties. By achieving a balance between exploration and exploitation through an ε-greedy strategy, the algorithm can efficiently approach the multi-objective Pareto front in dynamic environments.

[0049] (2) Reward Mechanism In the Q-learning enhanced neighborhood search algorithm implemented in this embodiment, the reward function is used to quantify the degree of quality improvement of the solution after applying a specific neighborhood operation. This enables Q-learning to adaptively select more effective neighborhood structures by learning feedback signals. Specifically, the reward value is defined as the normalized sum of all objective function improvements. Furthermore, a scaling factor is introduced in high-quality regions where the objective value is close to the current optimal solution to improve the algorithm's sensitivity to small improvements, thereby promoting the algorithm's search and convergence near the Pareto front.

[0050] The proposed version addresses the multi-objective minimization problem, where the reward function is defined as: (32) in This represents the initial solution. This represents the improved solution; The initial solution is represented in the th case. The target value on each target, The improved solution is represented in the th... The target value on each target; Indicates the number of the current population The minimum value of each objective. Indicates the number of the current population The maximum value of each target.

[0051] (3) Improved QVNS program This embodiment designs an improved generalized variable neighborhood search framework that integrates perturbation strategies and local search, with adaptive control via reinforcement learning. In the perturbation phase, an adjustable-strength destruction-reconstruction operator is employed to explore the solution space through random artifact deletion and heuristic reconstruction. The local search phase includes five specialized operators. The entire search process is driven by a two-layer Q-learning policy and evaluated using a normalized reward function based on multi-objective improvements. This design enables the algorithm to dynamically adjust its search behavior according to the search state, effectively improving convergence efficiency while maintaining diversity.

[0052] Optionally, S30 includes the following steps: S31, Based on the first Q table, determine the target damage intensity corresponding to the current first state; S32, perform the destruction-reconstruction operation corresponding to the target destruction intensity, remove and re-insert the corresponding number of workpieces from the current solution's workpiece sequence; S33, Based on the second Q table, determine the target local search operator type corresponding to the current second state; S34, Execute the selected local search operator to perform local optimization on the current solution; S35, calculate the normalized reward value of the multi-objective improvement, and update the first Q table and / or the second Q table, wherein the formula for calculating the normalized reward value is:

[0053] in, This represents the initial solution. This represents the improved solution; The initial solution is represented in the th case. The target value on each target, The improved solution is represented in the th... The target value on each target; Indicates the number of the current population The minimum value of each objective. Indicates the number of the current population The maximum value of each target.

[0054] Optionally, in step S34, the type of local search operator is controlled by the second Q-table (i.e., Q-table 2). The state space S2 is defined as the search operator type index, containing five operators: key swap (CSwap), key insertion (CInsr), key reversal (CInv), three-point permutation (TPP), and three-segment permutation (TSP). The action space A2 is also defined as the switching actions between operators, and Q-table 2 is a 5×5 matrix.

[0055] The specific operation methods of the five local search operators are as follows: The Critical Swap operator (CSwap) is used to identify and swap the positions of critical and non-critical workpieces. A critical workpiece is defined as the workpiece that has the greatest impact on scheduling performance in at least one objective dimension. It is determined by calculating the weighted sum of the processing time and setup time for each workpiece at each stage; the top 20% of workpieces with the largest weighted sum are marked as critical workpieces. During the operation, a critical workpiece and a non-critical workpiece are randomly selected, and their positions in the sequence are swapped.

[0056] The critical insertion operator (CInsr) removes the selected critical artifact from its current position and inserts it after a non-critical artifact. The selection of the insertion position is based on a local greedy criterion: for each possible insertion position, the changes in the sequence across the three objectives after insertion are calculated, and the position that maximizes the non-dominant improvement of the objectives is selected.

[0057] The critical inversion operator (CInv) selects a sequence segment between critical and non-critical workpieces (i.e., all workpieces from the current position of the critical workpiece to the current position of the non-critical workpiece) and reverses the overall order of the workpieces within that segment.

[0058] The three-point permutation operator (TPP) randomly selects three adjacent gene positions (i.e., three consecutive job indices) in the chromosome sequence, enumerates all six permutations of these three jobs, and selects the permutation that makes the overall three goals optimal to replace the original order.

[0059] The Three-Segment Permutation Operator (TSP) randomly selects two cut points on the chromosome sequence, dividing the sequence into three consecutive segments: the beginning, the middle, and the end (each segment contains at least one workpiece). It then enumerates all six possible permutations of the overall order of the three segments and selects the optimal permutation.

[0060] During the local search, if an operator fails to produce an improved solution (i.e., the new solution is not dependent on the old solution), the algorithm considers this attempt invalid and automatically switches to the next operator to continue trying, up to a maximum of three consecutive attempts. If all three attempts are invalid, the local search step is skipped, and the current solution remains unchanged. This backoff mechanism ensures that the algorithm does not consume excessive computational resources on unimproved local searches.

[0061] S40, calculate the adaptive crowding distance of each initial population, and use the adaptive crowding distance as a screening index to select the next generation population through fast non-dominated sorting and elite retention strategy; In this step, we select the next generation population from the combined set of parent and offspring individuals.

[0062] Optionally, the current parent population is first merged with the offspring population generated after the S30 evolution operation to obtain a temporary population. Then, fast non-dominated sorting is performed on the temporary population.

[0063] Fast Non-Dominated Sorting (FNS) is used to hierarchically classify solutions in a population based on dominance relationships. Its core idea is: for a minimization problem, a solution is considered dominant if it is no worse than any other solution on all objectives and is strictly better than any other solution on at least one objective. The FNS algorithm efficiently identifies each Pareto front through a two-stage iterative process: First, it calculates the dominance count for each solution; solutions not dominated by any other solution are assigned to the first front. Then, these solutions are temporarily removed, and the dominance counts of the remaining solutions are updated to identify the next front. This process iterates until all solutions are classified into a single non-dominated front. The specific process is as follows: Figure 3 As shown. Each non-dominated front has the following properties:

[0064]

[0065]

[0066] In the method for calculating crowding distance, the target values ​​within each Pareto front are first normalized. Then, all solutions are sorted according to each target dimension. Solutions at the boundaries are assigned infinite crowding distances, while solutions in the middle are normalized by calculating the sum of the absolute differences between their adjacent solutions and dividing by the range of the target value. The adaptive crowding distance for each solution is obtained by summing the distances across all target dimensions. Compared to traditional span-based calculation methods, this method focuses more on the local spacing between adjacent solutions, thus effectively improving the uniformity of population distribution in the target space while maintaining algorithm efficiency. See the formula for details:

[0067] in: Indicates the first The total crowding distance of each solution; This represents the total number of objective functions; Indicates the index of the current solution; , They respectively represent the first After sorting the targets, compare them with the current solution. The immediately preceding and following solutions; Indicates the first The solution is at the th solution. Function values ​​on each objective function; This indicates that in the current population, the [number]th ... The maximum value of each objective function; This indicates that in the current population, the [number]th ... The minimum value on an objective function.

[0068] After obtaining the current generation's scheduling scheme, S50 determines whether the target machine tool in the idle period should perform a shutdown-restart operation based on the energy consumption balance point; After obtaining the current generation scheduling scheme, this step performs intelligent decision-making for energy consumption optimization during the idle periods between different processes on each machine tool. The idle period is defined as the interval between the completion time of the previous workpiece and the start time of the next workpiece on the same machine tool. Traditionally, the machine tool is kept in standby mode during this interval, leading to unnecessary energy waste.

[0069] This embodiment introduces the concept of Energy Balance Point (EP), which is physically defined as the critical idle time when the additional electrical energy consumed by shutting down and restarting the machine tool is exactly equal to the electrical energy consumed by keeping it in standby mode during that period.

[0070] In some alternative implementations, the energy balance point (EP) is calculated using the following formula to determine whether a shutdown-restart strategy should be adopted based on the length of time the machine tool is idle.

[0071]

[0072]

[0073]

[0074] in It is a machine tool The energy consumption balance point is defined as the critical value of idle time when the energy cost of restarting is equal to the energy cost of keeping idle. It is the basic equilibrium point; high-speed machine tools, due to their large start-up losses and high production efficiency, The value is usually higher, and frequent start-stop cycles can actually reduce productivity; low-speed machine tools have lower start-up losses and relatively higher idle energy consumption. The values ​​are usually lower, so energy saving is a priority; Indicates workpiece In the stage By speed level Processed; Indicates the first Stage machine tools Above, workpiece With workpiece The free time between; It is a workpiece In the Completion time of each stage; It is a workpiece In the Completion time of each stage; Indicates the first Stage workpiece With workpiece Set the time between; Indicates the first Stage machine tools Restart time; It is an adjustment coefficient that adjusts the energy consumption balance point according to the processing speed level.

[0075] Optionally, the S50 includes: When the predicted idle time of the target machine tool is greater than the energy consumption balance point, the target machine tool is controlled to perform a shutdown-restart operation; When the predicted idle time is less than or equal to the energy consumption balance point, the target machine tool is controlled to remain in standby mode.

[0076] For example, suppose machine tool No. 2 in stage 2 adopts a shutdown-restart strategy, and its idle energy consumption per unit time is 2 (i.e., The restart energy consumption is 4 ( The energy consumption of shutting down is 0. In this example, on machine tool 2 in stage 2, the idle time between workpiece 5 and workpiece 3 is 7 units. If the machine is kept idle, the total idle energy consumption will be 14. However, if a shutdown-restart strategy is adopted, and assuming that the restart time of all machine tools is 1 unit, then the restart energy cost is 4, and the shutdown energy cost is 0, thus effectively reducing the overall energy consumption.

[0077] S60, in response to a newly arrived workpiece detected during production, identifies and locks completed and / or in-process workpieces and inherits their scheduling state, and merges the unstarted workpieces with the newly arrived workpieces for local rescheduling.

[0078] In real-world production environments, new workpieces may arrive randomly at any time, requiring the scheduling system to respond quickly without interrupting already started processing. This embodiment employs an event-triggered partial rescheduling mechanism, avoiding the extensive redundant calculations of full rescheduling through state layering and inheritance.

[0079] Optionally, event-triggered detection is implemented through the input queue of the MES system. The scheduling server polls the MES system's pending event queue every 30 seconds to check for a "new workpiece arrived" event. If the queue is empty, it continues to wait for the next detection cycle. If a new workpiece arrival event is detected, the system extracts the event's parameters: the number of new workpieces and the processing parameters for each new workpiece.

[0080] In the technical solution provided in this embodiment, a multi-objective optimization model is established with total energy consumption cost, carbon trading cost, and total weighted advance and delay penalties as optimization objectives; an evolutionary operation is performed on the current population based on Q-learning-driven adaptive variable neighborhood search; during the idle periods of machine tools in the scheduling scheme, the machine tools are decided to perform a shutdown-restart operation or remain in standby state based on the energy consumption balance point; in response to the arrival of a new workpiece, partial rescheduling is performed, inheriting the scheduling states of completed and currently being processed workpieces, and merging unstarted workpieces with newly arrived workpieces for local rescheduling.

[0081] Example 2 Building upon the first embodiment, this embodiment introduces a stability index to quantify the disturbance caused by rescheduling to the production system. This metric considers both the shift in workpiece start time and changes in machine tool allocation at different stages, thus reflecting the impact of rescheduling on the machining system. The expression for calculating the stability index is:

[0082] in, .

[0083] For example, suppose the start time of workpiece 5 in the first stage changes from 13 to 20, resulting in a deviation of 7; the start time of workpiece 5 in the second stage changes from 22 to 29, also resulting in a deviation of 7. The machine tool assigned to workpiece 6 in the second stage changes from S2-3 to S2-1, while the machine tool assignments for other workpieces remain unchanged. The specific stability calculation is as follows:

[0084]

[0085] Example 3 Based on the embodiments described above, in this embodiment, a complete scheduling decision in the scheduling problem includes the following three parts: (1) machine tool allocation at each stage; (2) workpiece sorting on each machine tool; and (3) shutdown and restart when the machine tool is idle. Traditional direct coding schemes usually encode machine tool allocation, workpiece sorting, and machine tool switching time between workpieces independently for each stage, which leads to a complex chromosome structure and high computational overhead.

[0086] This embodiment employs a compact and dynamically expandable encoding scheme, which encodes only the workpiece processing sequence in the first stage, representing it as a workpiece arrangement. Where n is the initial number of workpieces. When the number of workpieces dynamically arrives during the production process... When a new workpiece is created, the code will be dynamically expanded to... This represents the processing sequence of all workpieces in the first stage, while the insertion position of new workpieces is determined by the rescheduling strategy. The coding diagram is shown below. Figure 4 As shown. This encoding method retains the simplicity of traditional single-stage permutation encoding while introducing a dynamic expansion mechanism, enabling the workpiece sequence to flexibly cope with uncertainties in the production process, and avoiding the computational burden brought about by complex multi-stage encoding schemes.

[0087] Furthermore, in the decoding process of this embodiment, during the initial scheduling phase, the system allocates the workpieces to the machine tool that will complete them earliest, based on the chromosome arrangement order. In subsequent phases, the system automatically generates a workpiece list based on the workpieces' completion times in the previous phase and repeats the above allocation process.

[0088] For dynamic scheduling, this embodiment innovatively proposes a local rescheduling decoding mechanism: When a new workpiece is detected, the system first identifies and locks those workpieces that started or were completed before the arrival time, inheriting their machine allocation and scheduling. Then, it merges the unprocessed workpieces with the newly arrived workpiece into a rescheduled set and invokes a special local decoding method to schedule only this set. Finally, it integrates the inherited processing state with the new scheduling result to form a complete and continuous dynamic scheduling scheme. This mechanism not only inherits the efficiency of traditional LS decoding but also significantly reduces the computational overhead of handling dynamic events through state inheritance and local rescheduling.

[0089] Specifically, for different stages The workpiece is processed in different stages, each stage representing a single operation. The preparation time for the workpiece varies between stages, and each stage is equipped with the same number of parallel machine tools. The SDST between workpieces is determined by a matrix. This indicates that the matrix satisfies and In the single-stage decoding process, preparation time... It is strictly incorporated into the calculation of the time window. Assuming that during the stage... machine tools The previous workpiece processed was The workpiece currently being prepared for processing is So, the workpiece The start time is recorded as It must satisfy This means that the workpiece Only in the previous stage Completed, and currently in the process of completion. The machine tool can only begin processing after it has completed the corresponding settings. Sufficient buffer time is maintained between consecutive stages. The system automatically expands when a new workpiece arrives dynamically. The matrix is ​​used to accommodate newly added workpieces.

[0090] The scheduling status of completed and in-process workpieces is fully inherited: that is, their machine tool allocation schemes, processing time schedules, and speed level selections remain unchanged and are not subject to rescheduling. The core basis of this inheritance strategy is that the processing activities that have already begun are physically irreversible, and forcibly changing the allocation will lead to actual production interruption.

[0091] Unstarted workpieces and newly arrived workpieces are merged into a rescheduled set. For this set, a local decoding method based on list scheduling (LS) and earliest completion time (ECT) rules is used for optimized scheduling. The list scheduling process is as follows: Workpieces in the rescheduled set are sequentially retrieved according to their order in the current scheduling scheme; for the first unprocessed stage of a workpiece, all available machine tools in that stage are traversed, and the estimated completion time after assigning the workpiece to each machine tool is calculated; the machine tool that will allow the workpiece to complete earliest is selected for assignment. In subsequent stages, a workpiece processing order list for that stage is automatically generated based on the workpiece's completion time in the previous stage, and the above machine tool allocation process is repeated.

[0092] The inherited processing status is integrated with the local rescheduling results to form a complete dynamic scheduling scheme. During the integration process, it is necessary to ensure that the inherited part and the rescheduling part are correctly connected on the time axis: the start time of the first workpiece in the rescheduling part must not be earlier than the completion time of the last process in the inherited part.

[0093] Example 4 In this embodiment, the proposed algorithm CD-ENH is evaluated through experiments using established performance metrics. The experimental setup is detailed below: (i) For workpiece Its delivery window (Asadzadeh et al., 2016) The definition is as follows: • Calculate the total processing time:

[0094] • Calculate the average total processing time:

[0095] According to Sort the data in ascending order and determine the groups to set the target delivery date:

[0096] Introduce a perturbation parameter To establish a delivery window:

[0097] (ii) The time-of-use electricity pricing function is as follows:

[0098] Other example data is shown in Table 4: Table 4. Other experimental setups

[0099] To evaluate the contribution of each improved component in the algorithm, ablation experiments were conducted on 18 test cases in this embodiment. The performance of the algorithm was compared under four progressively enhanced variants: (1) NSGA-II-based initialization, (2) with the addition of a hybrid initialization strategy (Init-Mix), (3) with the addition of a multi-objective crossover operator (MOC), and (4) with the addition of DQ-NSGA-II (CD-ENH in this method) to improve crowding. For each variant and each test case, the HV and IGD indices were averaged over 10 independent runs, as summarized in Table 5. The box plots of HV and IGD values ​​are shown below. Figure 5 and Figure 6 As shown.

[0100] Table 5. Ablation Experiment Results

[0101] One-way ANOVA results showed highly significant differences between the groups (F = 38.03, p < 0.001; F = 20.00, p < 0.001), indicating that adding these four components sequentially significantly improves the algorithm's performance. Furthermore, Table 6 records the number of non-dominated solutions obtained for each subsequent variant, clearly demonstrating the superiority of the enhanced CD-ENH in terms of solution set diversity.

[0102] Table 6. NDS Results

[0103] Furthermore, this embodiment also proposes a hybrid flow shop rescheduling system, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the Q-learning-based hybrid flow shop rescheduling method described above.

[0104] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0105] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0109] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0110] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A hybrid flow shop rescheduling method based on Q-learning, characterized in that, The method includes the following steps: S10, Establish a multi-objective optimization model, wherein the multi-objective optimization model takes total energy consumption cost, carbon trading cost and total weighted advance and delay penalties as optimization objectives, and sequence-related setting time, machine tool speed level, processing sequence and energy consumption as constraints. S20, an initial population is generated using a hybrid initialization strategy. The hybrid initialization strategy consists of a heuristic solution of a first proportion and a random solution of a second proportion, and includes a sequential crossover operator, a cyclic crossover operator, and a sequence-based crossover operator. S30, based on Q-learning-driven adaptive variable neighborhood search, perform iterative updates on the initial population; S40, calculate the adaptive crowding distance of each initial population, and use the adaptive crowding distance as a screening index to select the next generation population through fast non-dominated sorting and elite retention strategy; After obtaining the current generation's scheduling scheme, S50 determines whether the target machine tool in the idle period should perform a shutdown-restart operation based on the energy consumption balance point; S60, in response to a newly arrived workpiece detected during production, identifies and locks completed and / or in-process workpieces and inherits their scheduling state, and merges the unstarted workpieces with the newly arrived workpieces for local rescheduling.

2. The hybrid flow shop rescheduling method based on Q-learning as described in claim 1, characterized in that, The objective function of the multi-objective optimization model described in S10 includes: (1)； In the formula, Indicates the total processing energy consumption of all machine tools. Indicates the total set energy consumption of all machine tools. This represents the total idle energy consumption of all machine tools. This indicates the total restart energy consumption of all machine tools; Indicates time period Electricity price; Indicates total energy consumption; This indicates the number of electricity price periods within a day; This indicates the total cost of carbon trading; Indicates the carbon emission coefficient per unit of energy consumption; (2)； In the formula, Indicates carbon emission quotas; Indicates the carbon trading price; (3)； In the formula, This indicates the total weighted average penalties for early and late payments; , Indicates workpiece Delayed completion penalty coefficient, workpiece Early completion penalty coefficient; , Indicates the earliest delivery date and the latest delivery date of the workpiece; This represents the completion time of workpiece j in the last stage. There are a total of g stages.

3. The hybrid flow shop rescheduling method based on Q-learning as described in claim 2, characterized in that, The constraints of the multi-objective optimization model described in S10 include: (1) Processing timing constraints: (4)； In the formula, Indicates the maximum completion time; N represents the set of workpieces; (5)； In the formula, For binary decision variables, if in stage workpiece Assigned to machine tools If the process is to be completed, then the value is 1; otherwise, it is 0. M represents the stage set. (6)； (7)； In the formula, For binary decision variables, if in stage machine tools Above, workpiece In the workpiece If it is a pre-processing step, take 1; otherwise, take 0. Indicates if in the stage machine tools If workpiece k is processed before workpiece j, then the value is 1; otherwise, the value is 0. Indicates if in the stage Workpiece k is assigned to the machine tool If the machining process is performed, then the value is 1; otherwise, it is 0. L represents the set of machine tools. (8)； In the formula, Indicates the number of machine tool speed classes; Represents a set of machine tool speed levels; Indicates if in the stage workpiece At speed level If it is being processed, take 1; otherwise, take 0. (9)； In the formula, Indicates the stage workpiece In machine tool The actual processing time; Indicates workpiece In the stage Standard processing time; Indicates the machine tool speed level The processing speed is reduced; (2) Time constraints: (10)； In the formula, Indicates workpiece In the stage Start processing time; Indicates workpiece In the stage Completion time; (11)； In the formula, Indicates workpiece The start processing time of stage i+1; (12)； In the formula, H represents a positive number; (13)； In the formula, Indicates the stage machine tool Start time of processing; Indicates that workpiece l is in stage Completion time; (14)； In the formula, Representation phase The setting time for machine tool l; Indicates the stage workpiece In the workpiece The setting time for the machining tool during subsequent processing; (15)； (16)； In the formula, This represents the total machine tool processing time across all stages; (17)； In the formula, Representation phase Total free time; Indicates the stage machine tool The time for stopping processing; Indicates the stage machine tool Start time of processing; (18)； In the formula, This represents the total idle time of the machine tool across all stages; (19)； In the formula, This indicates the total machine tool restart time across all stages; (20)； (3) Energy consumption constraints (21)； In the formula, Indicates the machine tool speed level Energy consumption per unit time of machine tool operation; (22)； (23)； In the formula, Indicates the stage Unit idle energy consumption; (24)； In the formula, Indicates the stage Unit restart energy consumption; (25)。 4. The hybrid flow shop rescheduling method based on Q-learning as described in any one of claims 1 to 3, characterized in that, The multi-objective optimization model also includes a stability index, the calculation expression of which is: ； in, 。 5. The hybrid flow shop rescheduling method based on Q-learning as described in claim 1, characterized in that, S30 includes: S31, Based on the first Q table, determine the target damage intensity corresponding to the current first state; S32, perform the destruction-reconstruction operation corresponding to the target destruction intensity, remove and re-insert the corresponding number of workpieces from the current solution's workpiece sequence; S33, Based on the second Q table, determine the target local search operator type corresponding to the current second state; S34, Execute the selected local search operator to perform local optimization on the current solution; S35, calculate the normalized reward value of the multi-objective improvement, and update the first Q table and / or the second Q table, wherein the formula for calculating the normalized reward value is: ； in, This represents the initial solution. This represents the improved solution; The initial solution is represented in the th case. The target value on each target, The improved solution is represented in the th... The target value on each target; Indicates the number of the current population The minimum value of each objective. Indicates the number of the current population The maximum value of each target.

6. The hybrid flow shop rescheduling method based on Q-learning as described in claim 1, characterized in that, The S50 includes: When the predicted idle time of the target machine tool is greater than the energy consumption balance point, the target machine tool is controlled to perform a shutdown-restart operation; When the predicted idle time is less than or equal to the energy consumption balance point, the target machine tool is controlled to remain in standby mode.

7. The hybrid flow shop rescheduling method based on Q-learning as described in claim 5, characterized in that, The formula for calculating the energy-saving balance point in S50 is: ； In the formula, It is a machine tool The energy consumption balance point is defined as the critical value of idle time when the energy cost of restarting is equal to the energy cost of keeping idle. It is the basic equilibrium point; Indicates workpiece In the stage By speed level Processed; It is an adjustment coefficient that adjusts the energy consumption balance point according to the processing speed level.

8. The hybrid flow shop rescheduling method based on Q-learning as described in claim 1, characterized in that, The adaptive congestion distance described in S40 is calculated using the following expression: ； In the formula, Indicates the first The total crowding distance of each solution; This represents the total number of objective functions; Indicates the index of the current solution; , They respectively represent the first After sorting the targets, compare them with the current solution. The immediately preceding and following solutions; Indicates the first The solution is at the th solution. Function values ​​on each objective function; This indicates that in the current population, the [number]th ... The maximum value of each objective function; This indicates that in the current population, the [number]th ... The minimum value on an objective function.

9. A hybrid assembly line workshop rescheduling system, characterized in that, The hybrid flow shop rescheduling system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the Q-learning-based hybrid flow shop rescheduling method as described in any one of claims 1 to 8.

Images