Multi-objective distributed permutation flow shop operation scheduling method based on q-learning

Abstract

The application discloses a multi-target distributed permutation flow shop operation scheduling method based on Q learning, relates to the technical field of workshop scheduling, and comprises the following steps: acquiring target parameters, preprocessing the target parameters to obtain preprocessing parameters, wherein the target parameters comprise equipment parameters and production task parameters; constructing a target mathematical model based on the preprocessing parameters, and generating three-layer encoding solutions comprising a workshop distribution layer, a workpiece sequencing layer and a machine speed selection layer based on the target mathematical model; and through the construction of the three-layer encoding solutions, the multi-heuristic initial population, the global search of the fusion particle swarm thought, the compound state perception and adaptive local search driven by Q learning, and the non-critical path optimization strategy, the scheduling scheme can better balance the production efficiency, energy consumption control and equipment maintenance demand as a whole.

Description

Technical Field

[0001] This invention relates to the field of workshop scheduling technology, and in particular to a multi-objective distributed permutation flow workshop operation and maintenance scheduling method based on Q-learning. Background Technology

[0002] Distributed replacement flow workshops are a typical production organization form for multi-factory collaborative manufacturing. In actual production, it is necessary to take into account production efficiency, green manufacturing and stable equipment operation. As manufacturing scenarios develop towards multi-factory, high-load and continuous operation, the collaborative optimization of workpiece scheduling, energy consumption control and equipment preventive maintenance has become an industry necessity. Intelligent operation and maintenance scheduling methods for this scenario have a wide range of application needs.

[0003] Existing distributed replacement flow shop scheduling methods mostly use completion time as the optimization objective. Some methods combine cultural gene algorithms to carry out global and local searches, relying on preset static rules to call local search operators. Some studies attempt to introduce reinforcement learning to assist decision-making, and output production and maintenance plans by constructing a scheduling model.

[0004] Existing scheduling models often employ idealized assumptions and fail to adequately consider the nonlinear coupling between machine productivity, total energy consumption, and equipment degradation. This makes it difficult to balance energy consumption and equipment health in multi-objective optimization. Local search operators rely on fixed rules and cannot adaptively adjust based on evolutionary stage, objective distribution, and equipment health, resulting in low search efficiency when handling multi-objective conflicts. Consequently, it is difficult to establish a stable and balanced scheduling scheme that balances productivity, total energy consumption, and equipment reliability. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multi-objective distributed permutation flow shop operation and maintenance scheduling method based on Q-learning, which solves the problem that existing scheduling models are unable to form a stable and balanced scheduling scheme among production efficiency, energy consumption, and equipment reliability.

[0006] The technical means employed in this invention are as follows:

[0007] In a first aspect, embodiments of the present invention provide a multi-objective distributed permutation workflow workshop operation and maintenance scheduling method based on Q-learning, including: Obtain target parameters, preprocess the target parameters to obtain preprocessed parameters, the target parameters including equipment parameters and production task parameters; A target mathematical model is constructed based on the preprocessing parameters, and a three-layer coded solution including a workshop allocation layer, a workpiece sorting layer, and a machine speed selection layer is generated based on the target mathematical model. An initial population is generated based on the three-layer coding solution and the device health status. A preliminary optimized population is obtained by performing a global evolutionary search on the initial population by incorporating particle swarm optimization. Based on the preliminary optimized population, a composite state space and an action space are constructed. The composite state space includes an evolutionary stage, a target space preference angle region, and machine health. The action space includes a maintenance perception operator. Based on the composite state space and the action space, a Q-learning agent interacts with the scheduling environment to obtain reward feedback and update the Q-table. Based on the updated Q-table, a PM-perception adaptive local search is performed on the preliminary optimized population to obtain a locally optimized population. A non-critical path optimization strategy is applied to the locally optimized population to obtain a finely optimized population; The refined optimization population is used as the initial population for the next iteration. The global evolutionary search, the PM-aware adaptive local search, and the non-critical path optimization strategy are repeatedly executed until the preset termination condition is met, and a scheduling scheme containing the refined optimization population is output.

[0008] Furthermore, the workshop allocation layer is used to determine the processing workshop to which the workpiece belongs, the workpiece sorting layer is used to determine the processing order of the workpiece within the workshop, and the machine speed selection layer is used to determine the processing speed of the workpiece on the corresponding machine.

[0009] Furthermore, the equipment health status includes the equipment's service life status; The generation of the initial population based on the three-layer encoded solution and the device health status includes: Based on the machine speed selection layer in the three-layer coding solution, the machine speed is reduced to the minimum to obtain the lowest energy consumption population; Based on the workshop allocation layer in the three-layer coding solution, the total processing time of the workpieces is accumulated, the processing time of each workpiece is sorted from largest to smallest, and the workpieces are evenly distributed to each factory to obtain the factory load balancing population. Based on the three-layer encoding solution, the scheduling solution is randomly initialized to obtain a random population; Based on the three-layer coding solution and the equipment service life status level, a service life status level population adapted to the equipment service life status is generated; The initial population is constructed based on the lowest energy consumption population, the factory load balancing population, the random population, and the service life status level population.

[0010] Furthermore, the step of performing a global evolutionary search on the initial population by incorporating particle swarm optimization techniques to obtain a preliminarily optimized population includes: Crossover and mutation operations are performed on the initial population to introduce the position update rules of the particle swarm into the evolutionary process. The position information of individuals in the population is updated according to the individual fitness. Through multiple generations of evolutionary iteration, the preliminary optimized population is obtained based on non-dominated sorting.

[0011] Furthermore, the maintenance-aware operator includes: executing a critical acceleration operator with maintenance look-ahead veto, a conservative deceleration operator for non-critical path idle time, a health transfer operator based on the remaining service life safety period of the target plant, and executing a critical block reorganization operator based on wear-out driven operation.

[0012] Furthermore, the step of applying a non-critical path optimization strategy to the local optimization population to obtain a refined optimization population includes: identifying critical and non-critical paths in the scheduling scheme, and adjusting the machine processing speed based on the relaxation time of the processes on the non-critical paths to reduce the total energy consumption of the equipment.

[0013] Secondly, embodiments of the present invention also provide a multi-objective distributed permutation workflow workshop operation and maintenance scheduling system based on Q-learning, including: The preprocessing module is used to acquire target parameters, preprocess the target parameters to obtain preprocessed parameters, wherein the target parameters include equipment parameters and production task parameters; The model building module is used to build a target mathematical model based on the preprocessing parameters, and generate a three-layer coded solution including a workshop allocation layer, a workpiece sorting layer and a machine speed selection layer based on the target mathematical model. The initial optimization population acquisition module is used to generate an initial population based on the three-layer encoding solution and the device health status, and to perform a global evolutionary search on the initial population by incorporating particle swarm optimization to obtain the initial optimized population; The local optimization population acquisition module is used to construct a composite state space and an action space based on the preliminary optimization population. The composite state space includes an evolutionary stage, a target space preference angle region, and machine health. The action space includes a maintenance perception operator. Based on the composite state space and the action space, a Q-learning agent interacts with the scheduling environment to obtain reward feedback and update the Q-table. Based on the updated Q-table, a PM-perception adaptive local search is performed on the preliminary optimization population to obtain a locally optimized population. The fine-grained optimization population acquisition module is used to execute non-critical path optimization strategies on the locally optimized population to obtain a fine-grained optimized population. The output module is used to take the finely optimized population as the initial population for the next iteration, repeatedly execute the global evolutionary search, the PM-aware adaptive local search, and the non-critical path optimization strategy until a preset termination condition is met, and output a scheduling scheme containing the finely optimized population.

[0014] Compared with the prior art, the present invention has the following advantages: This invention enhances the diversity of solutions and the quality of the starting point for optimization by constructing a three-layer encoded solution and a multi-heuristic initial population; it strengthens the global exploration capability of the algorithm by incorporating global search with particle swarm optimization; it enables the optimization strategy to dynamically adapt to the equipment health status and multi-objective preferences by using Q-learning-driven composite state perception and adaptive local search; and it reduces equipment energy consumption without affecting the overall completion time by using a non-critical path optimization strategy, thus enabling the scheduling scheme to better balance production efficiency, energy consumption control, and equipment maintenance needs.

[0015] Based on the above reasons, this invention can be widely applied in fields such as workshop scheduling. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the process of the present invention; Figure 2 This is a flowchart of the Q-SAMA algorithm of the present invention; Figure 3 This is a diagram showing the selection of the global optimal solution in this invention; Figure 4 To reinforce the learning process diagram; Figure 5 This is a diagram illustrating the non-critical path-oriented speed reduction strategy for total energy consumption in this invention. Figure 6 This is a diagram illustrating the lossless time optimization strategy for key block reassembly in this invention. Figure 7 The experimental results of the algorithm are shown in the figure; Figure 8 For comparison of the HV box plots of the algorithms; Figure 9 The Pareto front solution set diagram is used for comparison. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0019] It should be noted that the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0020] This invention provides a multi-objective distributed permutation flow shop operation and maintenance scheduling method based on Q-learning, which is used to solve the problem that existing scheduling models are unable to form a stable and balanced scheduling scheme between production efficiency, energy consumption and equipment reliability.

[0021] The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0022] Please see Figure 1 , Figure 1 This is a schematic diagram of the process of the present invention.

[0023] This application provides a multi-objective distributed permutation flow shop operation and maintenance scheduling method based on Q-learning, characterized by the following steps: Step 101: Obtain the target parameters and preprocess them to obtain preprocessed parameters. The target parameters include equipment parameters and production task parameters. Equipment parameters and production task parameters can reflect the production constraints and equipment operating status of the distributed replacement flow workshop. Preprocessing the target parameters can standardize the data format and unify the units, thereby improving the accuracy of constructing the multi-objective scheduling mathematical model.

[0024] Step 102: Construct a target mathematical model based on preprocessing parameters, and generate a three-layer coded solution based on the target mathematical model, including a workshop allocation layer, a workpiece sorting layer, and a machine speed selection layer. This three-layer coded solution can completely map the scheduling logic of a multi-objective distributed replacement flow shop, improving the algorithm's adaptability to complex scheduling scenarios.

[0025] In the multi-objective distributed permutation flow shop scheduling problem, job allocation, sorting, and machine speed selection decisions are highly coupled, resulting in a complex and discrete solution space. This application proposes a three-layer encoding mechanism consisting of a factory allocation layer, a job sorting layer, and a machine speed layer. This mechanism maps the complex physical scheduling logic into a data structure that facilitates algorithm computation, thereby improving the algorithm's expressive power and optimization efficiency.

[0026] For example, taking the shop floor scheduling problem involving 2 factories, 4 workpieces, 4 machines and 3 speed levels as an example, the coding of one solution is shown in Tables 1 and 2 below.

[0027] Table 1. Workpiece Sorting and Factory Assignment Codes

[0028] Job sorting layer: Employs a full permutation encoding based on job indexes. This layer is a vector of length n, representing the global priority sequence of all jobs before they enter the factory for allocation. For example, the sequence [2,3,4,1] indicates that job 3 has the highest priority of 4.

[0029] Factory allocation layer: Uses an integer vector of length n, where each bit ranges from [1, f], representing the factory number assigned to the corresponding workpiece. For example, if the allocation vector is [1, 2, 1, 2], combined with the workpiece sorting layer, it means that workpieces 2 and 4 are assigned to factory 2, and workpieces 1 and 3 are assigned to factory 1.

[0030] Table 2 Processing Speed ​​Code

[0031] Machine speed layer: using an n A matrix of size m represents the processing speed selection for each workpiece on each machine. The elements v in the matrix... j,i This represents the speed level of workpiece j when it is processed on machine i. For example, element 2 in the first row and first column indicates that the speed level of workpiece j on machine i is 2. This layer determines the actual processing time and processing energy consumption of the workpiece.

[0032] In some embodiments, the workshop allocation layer is used to determine the processing workshop to which a workpiece belongs, the workpiece sorting layer is used to determine the processing order of the workpiece within the workshop, and the machine speed selection layer is used to determine the processing speed of the workpiece on the corresponding machine. The workshop allocation layer assigns workpieces to various workshops, the workpiece sorting layer determines the processing order of workpieces within each workshop, and the machine speed layer selects the processing speed of each process. This three-layer encoding solution can completely cover all dimensions of scheduling decisions, facilitating efficient iteration and optimization of different scheduling schemes by the algorithm.

[0033] Step 103: Generate an initial population based on the three-layer encoded solution and equipment health status. Perform a global evolutionary search on the initial population by incorporating particle swarm optimization (PSO) techniques to obtain a preliminary optimized population. Combining the three-layer encoded solution with equipment health status information to construct the initial population helps generate an initial solution that better reflects the actual working conditions. Incorporating PSO techniques for a global evolutionary search expands the solution space exploration range while yielding a higher-quality preliminary optimized population.

[0034] Integrating the particle swarm optimization concept means borrowing the mechanism in the particle swarm optimization algorithm where individuals follow the group's optimal position and update their own historical optimal position, guiding the population to move towards high-quality solution regions, improving the algorithm's ability to explore the global solution space, and making the resulting preliminary optimized population have better diversity and convergence.

[0035] In some embodiments, the equipment health status includes the equipment service life status; generating an initial population based on the three-layer coding solution and the equipment health status includes: based on the machine speed selection layer in the three-layer coding solution, reducing the machine speed to the minimum to obtain the minimum energy consumption population; based on the workshop allocation layer in the three-layer coding solution, accumulating the total processing time of the workpieces, sorting the processing time of each workpiece from largest to smallest, and evenly distributing the workpieces to various factories to obtain the factory load balancing population; based on the three-layer coding solution, randomly initializing the scheduling solution to obtain the random population; based on the three-layer coding solution and the equipment service life status level, generating a service life status level population adapted to the equipment service life status; and constructing the initial population based on the minimum energy consumption population, the factory load balancing population, the random population, and the service life status level population.

[0036] By generating a minimum energy consumption population, a factory load balancing population, a random population, and a service life status population, the initial population can simultaneously contain scheduling solutions with low energy consumption, load balancing, high randomness, and appropriate equipment service life status. This can effectively improve the diversity of the initial population, provide a solution space covering multiple high-quality features for subsequent evolutionary search, and help improve the starting quality of the algorithm's optimization.

[0037] In some embodiments, a preliminary optimized population is obtained by incorporating particle swarm optimization (PSO) principles to perform a global evolutionary search on the initial population. This includes: performing crossover and mutation operations on the initial population; introducing the PSO position update rules into the evolutionary process; updating the position information of individuals in the population based on their fitness; iterating through multiple generations of evolution; and obtaining the preliminary optimized population based on non-dominated sorting. Introducing the PSO position update rules into the crossover and mutation operations guides individuals in the population to update their positions by referring to their own fitness information and that of the group. This continuously optimizes the quality of the solution through multiple generations of evolutionary iteration, helping to improve the algorithm's global exploration capability and convergence efficiency, resulting in a higher-quality preliminary optimized population.

[0038] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the selection of the global optimal solution in this invention.

[0039] Specifically, by introducing the optimization-oriented approach of the Particle Swarm Optimization (PSO) algorithm, the improved operator selects solutions with larger crowding distances from the Pareto solution set as the global best (Gbest), thereby guiding the population to approach the two ends and gaps of the Pareto front, which helps maintain the diversity of solution distribution. This process mainly includes the following three layers of operations: Factory layer crossover and mutation: Using dynamically changing nonlinear probabilities, the allocation of workpieces in the current solution is adjusted to converge with the factory corresponding to the best solution, or the factories are randomly reassigned to workpieces. This method can both learn the characteristics of the best allocation scheme and maintain a certain ability to explore the solution space, avoiding premature convergence of the population.

[0040] Job layer crossover and mutation: The Partially Mapped Crossover (PMX) method is used to align the relative order of jobs in the current solution with the best Gbest solution and repair it to a feasible solution. Specifically, the positions of two jobs in the current solution are swapped. This operation can efficiently absorb the ordering information of high-quality solutions while maintaining the validity of the sequence, thus improving the optimization efficiency of the scheduling scheme.

[0041] Machine speed layer crossover and mutation: The machine speed of the current solution is compared with the machine speed corresponding to Gbest, and a trending operation is performed: if the current speed is less than the speed corresponding to Gbest, an upshift operation is likely to be performed; otherwise, a downshift operation is likely to be performed. This operation allows the machine speed configuration to dynamically approach the optimal solution, achieving a better balance between processing efficiency and energy consumption control.

[0042] Step 104: Construct a composite state space and action space based on the initial optimized population. The composite state space includes the evolutionary stage, the target space preference angle region, and the machine health. The action space includes the maintenance perception operator. Based on the composite state space and action space, the Q-learning agent interacts with the scheduling environment to obtain reward feedback and update the Q-table. Based on the updated Q-table, perform PM perception adaptive local search on the initial optimized population to obtain the locally optimized population.

[0043] By constructing a composite state space that includes evolutionary stages, target space preference angles, and machine health, and defining the action space with a maintenance perception operator, the Q-learning agent can learn a more suitable local search strategy in its interaction with the scheduling environment. Based on the updated Q-table, PM perception adaptive local search can be performed to optimize the initial optimization population in a targeted manner, improving the adaptability of the local optimization population to multi-objective scheduling scenarios and equipment health status.

[0044] Please see Figure 4 , Figure 4 To reinforce the learning process diagram.

[0045] By embedding Q-learning as an intelligent agent within the algorithm, each scheduling solution and its associated machine age information are considered the scheduling environment. Through continuous interaction with this environment, the agent autonomously learns and establishes a "state-action-reward" mapping relationship. This enables intelligent adaptive scheduling at different evolutionary stages, based on target preferences and equipment health states, thereby improving the scheduling scheme's adaptability to complex operating conditions. The learning process is as follows: I. Constructing a Composite State Space: The state space must comprehensively characterize the core features of the current scheduling solution. This invention defines a composite state S = {St_evolve, St_angle, St_health}. Wherein, St_evolve (evolutionary stage): reflects the iterative progress of the global search. It is divided into three levels: early stage (0-33% of maximum iterations), mid-stage (33%-66% of maximum iterations), and late stage (66%-100% of maximum iterations). This division allows the agent to dynamically adjust its strategy according to the search progress, avoiding premature convergence in the early stage or insufficient search efficiency in the later stage; St_angle (preference angle region): reflects the position of the solution in the target space. It is divided into three levels: large angle (biased towards completion time optimization), medium angle (balancing completion time and energy consumption optimization), and small angle (biased towards energy consumption optimization). This dimension can guide the agent to flexibly switch optimization priorities among multiple objectives, better meeting the scheduling needs of different scenarios; St_health (system health status): reflects the maintenance pressure of the current solution, defined as the percentage of the average service life of all machines in the current solution relative to the maintenance threshold D_PM. It is divided into two levels: high health (less than 60%, indicating that the machine is in good condition and is unlikely to trigger maintenance) and low health (greater than or equal to 60%, on the verge of major repair and very likely to trigger maintenance). This dimension allows the intelligent agent to perceive equipment maintenance risks in advance and reduce the probability of production interruption.

[0046] 2. Construct the action space by extracting four maintenance perception operators as action options.

[0047] In some embodiments, the maintenance-aware operator includes: executing a critical acceleration operator with maintenance look-ahead veto, a conservative deceleration operator for non-critical path idle time, a health transfer operator based on the target plant's remaining service life safety period, and executing a critical block reorganization operator based on wear and tear. Through the combined action of the critical acceleration operator, the conservative deceleration operator, the health transfer operator, and the critical block reorganization operator, the processing efficiency of the critical path can be improved while reasonably controlling equipment energy consumption and service life loss, thus balancing scheduling target optimization and equipment health maintenance needs.

[0048] Third, the reward function is designed using a two-dimensional approach, combining basic rewards and penalties with maintenance-related additional rewards. The basic reward and penalty rules are as follows: The scheduling scheme is evaluated based on Pareto dominance. If the new solution strictly dominates the old solution, a positive feedback reward of +10 is given; if the new solution and the old solution do not dominate each other, a positive feedback reward of +5 is given; if the new solution is dominated by the old solution, a negative feedback penalty of -5 is given. This design guides the agent towards the Pareto front, improving the multi-objective performance of the scheduling scheme.

[0049] Maintenance Bonus: If an agent's actions reduce the total number of maintenance operations triggered by the system, an additional positive reward is given; if the actions increase the number of maintenance operations, a penalty is imposed. This bonus enhances the agent's focus on equipment health, optimizing scheduling goals while reducing equipment maintenance costs and downtime risks.

[0050] IV. Q-value update: The agent selects actions based on a preset policy to generate a new scheduling solution. The preset policy can be... The strategy is developed, and the Q-values ​​of the corresponding "state-action" pairs are updated using the Bellman equation in dynamic programming. Through continuous iterative updates, the Q-table can gradually learn the optimal action selection strategy for different states, making the agent's local search process more aligned with the multi-objective requirements and device health constraints of the scheduling scenario, ultimately resulting in a locally optimized population that balances scheduling performance and device maintenance needs.

[0051] Step 105: Apply non-critical path optimization strategies to the locally optimized population to obtain a finely optimized population. Applying non-critical path optimization strategies to the locally optimized population can further explore the scheduling optimization space of non-critical processes, reduce idle time and energy waste of non-critical paths, and improve the overall performance of the scheduling scheme without affecting the overall production cycle, resulting in a finely optimized population that better meets actual production needs.

[0052] In some embodiments, a non-critical path optimization strategy is applied to the local optimization population to obtain a refined optimization population. This includes: identifying critical and non-critical paths in the scheduling scheme, and adjusting the machine processing speed based on the relaxation time of the processes on the non-critical paths to reduce the total energy consumption of the equipment. This approach, without affecting the processing progress of the critical path, fully utilizes the relaxation time of the non-critical path processes. By reasonably adjusting the machine processing speed, it can effectively reduce the total energy consumption of the equipment while minimizing unnecessary equipment wear and tear, resulting in a refined optimization population that balances completion time and energy consumption optimization objectives.

[0053] Step 106: Using the refined optimization population as the initial population for the next iteration, repeatedly execute the global evolutionary search, PM-aware adaptive local search, and non-critical path optimization strategies until the preset termination condition is met, and output a scheduling scheme containing the refined optimization population. Using the refined optimization population as the initial population for the next iteration allows the algorithm to continue iterative optimization based on the already optimized solution. Combining a multi-stage strategy of global exploration and local refinement, it gradually approaches a better Pareto front, and the final output scheduling scheme achieves a good balance between multi-objective performance and equipment maintainability.

[0054] This invention enhances the diversity of solutions and the quality of the starting point for optimization by constructing a three-layer encoded solution and a multi-heuristic initial population; it strengthens the global exploration capability of the algorithm by incorporating global search with particle swarm optimization; it enables the optimization strategy to dynamically adapt to the equipment health status and multi-objective preferences by using Q-learning-driven composite state awareness and adaptive local search; and it reduces equipment energy consumption without affecting the overall completion time by using a non-critical path optimization strategy, thus enabling the scheduling scheme to better balance production efficiency, energy consumption control, and equipment maintenance needs.

[0055] This invention also provides a multi-objective distributed permutation flow shop operation and maintenance scheduling system based on Q-learning, comprising: a preprocessing module for acquiring target parameters, preprocessing the target parameters to obtain preprocessed parameters, the target parameters including equipment parameters and production task parameters; a model building module for constructing a target mathematical model based on the preprocessed parameters, and generating a three-layer coded solution including a shop allocation layer, a workpiece sorting layer, and a machine speed selection layer based on the target mathematical model; a preliminary optimization population acquisition module for generating an initial population based on the three-layer coded solution and equipment health status, and performing a global evolutionary search on the initial population by incorporating particle swarm optimization to obtain a preliminary optimization population; and a local optimization population acquisition module for constructing a composite state space and actions based on the preliminary optimization population. The system consists of a composite state space (including evolutionary stages, a target space preference angle region, and machine health) and an action space (including maintenance perception operators). Based on the composite state space and action space, a Q-learning agent interacts with the scheduling environment to obtain reward feedback and update the Q-table. The updated Q-table is then used to perform a PM-perception adaptive local search on the initial optimized population, resulting in a locally optimized population. A refined optimized population acquisition module is used to execute non-critical path optimization strategies on the locally optimized population, resulting in a refined optimized population. An output module uses the refined optimized population as the initial population for the next iteration, repeatedly executing global evolutionary search, PM-perception adaptive local search, and non-critical path optimization strategies until a preset termination condition is met, outputting a scheduling scheme containing the refined optimized population.

[0056] Please see Figure 2 , Figure 2 This is a flowchart of the Q-SAMA algorithm of the present invention.

[0057] The specific process of the Q-SAMA algorithm is as follows: the scheduling scheme is initialized using the step-decreasing initialization method to generate initial population 1. This step can combine the service life status of the equipment to generate an initial solution that adapts to the actual working conditions, thereby improving the diversity and quality of the initial population.

[0058] Based on the initial population 1, a global evolution operation based on the particle swarm optimization concept is performed to obtain population 2. This step can guide the population to converge towards the high-quality solution region, enhance the algorithm's ability to explore the global solution space, and improve the overall optimization level of the scheduling scheme.

[0059] Population 2 executes a maintenance-aware local search strategy based on reinforcement learning, and then filters according to dominance and crowding to obtain population 3. This step can adaptively adjust the search strategy according to the evolutionary stage, target preference and equipment health, thereby improving the accuracy of local optimization and reducing the probability of equipment maintenance being triggered.

[0060] Population 3 is merged with the obtained Pareto front solution set, and a lossless energy and time optimization strategy is performed on the merged solution set to obtain population 4. This step can make full use of the relaxation time of non-critical paths to reduce energy consumption and further improve the overall performance of the solution without affecting the completion time.

[0061] Determine whether the preset iteration termination condition is met. If not, return population 4 as the new initial population to continue the iteration. If the condition is met, select the Pareto optimal solution set from population 4 and output the final scheduling scheme. This step continuously improves the quality of the solution through iterative iteration, so that the final scheme can better balance production efficiency, energy consumption and equipment reliability.

[0062] Please see Figure 5 , Figure 5 This is a diagram of the non-critical path-oriented speed reduction strategy for total energy consumption in this invention.

[0063] This invention employs a lossless energy reduction strategy based on idle time. Due to the flow and waiting between processes in a production line, machines often idle, resulting in wasted energy. Taking into account the physical characteristic that higher machine speeds generate higher power, this invention designs a precise speed reduction mechanism for non-critical paths. The specific steps are as follows: First, accurately locate the idle window: In the Gantt chart of the current scheduling scheme, identify non-critical processes located on non-critical paths that subsequently involve "machine idling" or "workpiece waiting" idle time. Then, reduce the machine processing speed of the locked process by one level; during this process, apply strict safety boundary constraints to ensure that the extended processing time due to the reduced speed does not exceed the original idle time margin of that process. This approach can fully utilize the relaxation time of non-critical path processes to reduce machine speed while ensuring the normal progress of the critical path and without extending the overall production cycle, effectively reducing total equipment energy consumption and unnecessary wear, and achieving a better balance between completion time, energy consumption control, and equipment health maintenance in the scheduling scheme.

[0064] Please see Figure 6 , Figure 6 This is a diagram illustrating the lossless time optimization strategy for key block reassembly in this invention.

[0065] The lossless time optimization strategy based on critical block reorganization first independently evaluates the actual processing time of each factory to accurately identify the critical factory that determines the system bottleneck. Then, it shields the interference of non-critical factories and performs a greedy insertion neighborhood search with an iterative upper limit only for the workpiece sequence within the critical factory. Finally, it maps the selected optimal local sequence back to the global population in situ, achieving lossless decoupling and precise compression of the system's maximum completion time under the constraint of equipment service life.

[0066] Please see Figure 7 , Figure 7 The experimental results of the algorithm are shown in the figure.

[0067] To verify the effectiveness of the proposed Q-SAMA algorithm, it was compared with four mainstream multi-objective optimization algorithms: Dual Population Algorithm (DPA), Multi-Objective Whale Swarm Algorithm (MOWSA), Improved Hybrid Tussock Swarm Algorithm (IHSSA), and Two-Stage Multi-Objective Evolutionary Algorithm (TEA). The experiments were conducted using 33 standard test cases with varying numbers of factories, machines, and workpieces, with hypervolume (HV) used as the performance evaluation metric.

[0068] This invention implements the proposed model and algorithm using MATLAB R2021b, with an Intel Core i9 5.4GHz processor, 32GB of RAM, and Windows 11 operating system. The hypervolume change curves obtained by each algorithm on 33 test cases are shown below. Figure 7 As shown in the figure. The experimental results show that the Q-SAMA algorithm proposed in this invention outperforms other comparative algorithms in terms of hypervolume index in each set of examples, which fully demonstrates that the algorithm has better solution quality and convergence performance.

[0069] Please see Figure 8 , Figure 8 The HV box plots are used for comparison of the algorithms.

[0070] To further verify the superiority of the algorithm of this invention, four typical test cases were selected, and hypervolume (HV) box plots of the five comparative algorithms were plotted under multiple independent runs. The results are as follows. Figure 8 As shown in the figure, the Q-SAMA algorithm proposed in this invention has the highest HV index in all examples, indicating that the Pareto front solution set it obtains has better convergence and distribution. At the same time, the bin height (interquartile range) of this algorithm is flatter, the data distribution is more compact, and there are no obvious outliers, indicating that the Q-SAMA algorithm can stably output high-quality scheduling solutions in multiple repeated runs, and has better robustness and operational stability.

[0071] Please see Figure 9 , Figure 9The Pareto front solution set diagram is used for comparison.

[0072] To more intuitively demonstrate the performance comparison of the various algorithm solutions, Pareto front solution sets of each algorithm on test cases were plotted.

[0073] The points marked with yellow asterisks in the figure represent the Pareto fronts obtained by the Q-SAMA algorithm of this invention, and are significantly distributed to the lower left of the other comparative algorithms. This indicates that, under the same maximum completion time, the Q-SAMA algorithm can obtain a scheduling scheme with lower total energy consumption; and under the same energy consumption level, it can provide a production schedule with a shorter completion time. The algorithm of this invention has a strong dominant effect on the other four comparative algorithms, demonstrating that it can generate scheduling solutions that combine engineering practicality and optimization quality when balancing the conflicting objectives of production efficiency and energy consumption.

[0074] Whether in the small-scale example DPFSP08 or the large-scale example DPFSP33, the Q-SAMA algorithm consistently maintains its leading advantage. This demonstrates that the algorithm's internal adaptive mechanism—the action policy learned in the state space by Q-learning—possesses excellent generalization ability and does not fall into local optima due to the exponential expansion of the search space.

[0075] In summary, the Q-SAMA algorithm outperforms other comparative algorithms in terms of both the HV evaluation metric and the Pareto front distribution. Its non-dominated solution set is significantly distributed in the lower left of the coordinate axis, enabling it to find a high-quality compromise between maximum completion time and total energy consumption. The algorithm highly concentrates search resources on the core high-quality compromise region, making reasonable trade-offs in the extreme breadth of the solution set, which closely aligns with the engineering requirements of actual factory production scheduling.

[0076] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0077] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-objective distributed permutation flow shop operation and maintenance scheduling method based on Q-learning, characterized in that, include: Obtain target parameters, preprocess the target parameters to obtain preprocessed parameters, the target parameters including equipment parameters and production task parameters; A target mathematical model is constructed based on the preprocessing parameters, and a three-layer coded solution including a workshop allocation layer, a workpiece sorting layer, and a machine speed selection layer is generated based on the target mathematical model. An initial population is generated based on the three-layer coding solution and the device health status. A preliminary optimized population is obtained by performing a global evolutionary search on the initial population by incorporating particle swarm optimization. Based on the preliminary optimized population, a composite state space and an action space are constructed. The composite state space includes an evolutionary stage, a target space preference angle region, and machine health. The action space includes a maintenance perception operator. Based on the composite state space and the action space, a Q-learning agent interacts with the scheduling environment to obtain reward feedback and update the Q-table. Based on the updated Q-table, a PM-perception adaptive local search is performed on the preliminary optimized population to obtain a locally optimized population. A non-critical path optimization strategy is applied to the locally optimized population to obtain a finely optimized population; The refined optimization population is used as the initial population for the next iteration. The global evolutionary search, the PM-aware adaptive local search, and the non-critical path optimization strategy are repeatedly executed until the preset termination condition is met, and a scheduling scheme containing the refined optimization population is output.

2. The multi-objective distributed permutation flow shop operation and maintenance scheduling method based on Q-learning according to claim 1, characterized in that, The workshop allocation layer is used to determine the processing workshop to which the workpiece belongs, the workpiece sorting layer is used to determine the processing order of the workpiece in the workshop, and the machine speed selection layer is used to determine the processing speed of the workpiece on the corresponding machine.

3. The multi-objective distributed permutation flow shop operation and maintenance scheduling method based on Q-learning according to claim 1, characterized in that, The equipment health status includes the equipment's service life status; The generation of the initial population based on the three-layer encoded solution and the device health status includes: Based on the machine speed selection layer in the three-layer coding solution, the machine speed is reduced to the minimum to obtain the lowest energy consumption population; Based on the workshop allocation layer in the three-layer coding solution, the total processing time of the workpieces is accumulated, the processing time of each workpiece is sorted from largest to smallest, and the workpieces are evenly distributed to each factory to obtain the factory load balancing population. Based on the three-layer encoding solution, the scheduling solution is randomly initialized to obtain a random population; Based on the three-layer coding solution and the equipment service life status level, a service life status level population adapted to the equipment service life status is generated; The initial population is constructed based on the lowest energy consumption population, the factory load balancing population, the random population, and the service life status level population.

4. The multi-objective distributed permutation flow shop operation and maintenance scheduling method based on Q-learning according to claim 1, characterized in that, The process of performing a global evolutionary search on the initial population by incorporating particle swarm optimization techniques to obtain a preliminary optimized population includes: Crossover and mutation operations are performed on the initial population to introduce the position update rules of the particle swarm into the evolutionary process. The position information of individuals in the population is updated according to the individual fitness. Through multiple generations of evolutionary iteration, the preliminary optimized population is obtained based on non-dominated sorting.

5. The multi-objective distributed permutation flow shop operation and maintenance scheduling method based on Q-learning according to claim 1, characterized in that, The maintenance-aware operators include: executing a critical acceleration operator with maintenance look-ahead veto, a conservative deceleration operator for non-critical path idle time, a health transfer operator based on the remaining service life safety period of the target plant, and executing a critical block reorganization operator based on wear-driven operation.

6. The multi-objective distributed permutation flow shop operation and maintenance scheduling method based on Q-learning according to claim 1, characterized in that, The step of performing a non-critical path optimization strategy on the local optimization population to obtain a refined optimization population includes: identifying critical paths and non-critical paths in the scheduling scheme, and adjusting the machine processing speed based on the relaxation time of the processes on the non-critical paths to reduce the total energy consumption of the equipment.

7. A multi-objective distributed permutation workflow workshop operation and maintenance scheduling system based on Q-learning, characterized in that, include: The preprocessing module is used to acquire target parameters, preprocess the target parameters to obtain preprocessed parameters, wherein the target parameters include equipment parameters and production task parameters; The model building module is used to build a target mathematical model based on the preprocessing parameters, and generate a three-layer coded solution including a workshop allocation layer, a workpiece sorting layer and a machine speed selection layer based on the target mathematical model. The initial optimization population acquisition module is used to generate an initial population based on the three-layer encoding solution and the device health status, and to perform a global evolutionary search on the initial population by incorporating particle swarm optimization to obtain the initial optimized population; The local optimization population acquisition module is used to construct a composite state space and an action space based on the preliminary optimization population. The composite state space includes an evolutionary stage, a target space preference angle region, and machine health. The action space includes a maintenance perception operator. Based on the composite state space and the action space, a Q-learning agent interacts with the scheduling environment to obtain reward feedback and update the Q-table. Based on the updated Q-table, a PM-perception adaptive local search is performed on the preliminary optimization population to obtain a locally optimized population. The fine-grained optimization population acquisition module is used to execute non-critical path optimization strategies on the locally optimized population to obtain a fine-grained optimized population. The output module is used to take the finely optimized population as the initial population for the next iteration, repeatedly execute the global evolutionary search, the PM-aware adaptive local search, and the non-critical path optimization strategy until a preset termination condition is met, and output a scheduling scheme containing the finely optimized population.

Images