Intelligent scheduling method and system for flexible workshop based on Q-learning enhanced hierarchical sparrow search

Abstract

The application belongs to the technical field of intelligent manufacturing and production scheduling, and particularly relates to a flexible workshop intelligent scheduling method and system based on Q-Learning enhanced layered sparrow search, which adopts GLR strategy and Tent chaotic mapping to respectively generate two sequences of MS and OS to obtain a high-quality initial scheduling scheme population, and specifically improves the diversity of the population, reduces the risk of the algorithm falling into local optimization, and enhances the overall optimization performance and convergence speed by performing layered updating on the population; the key process concept composed of non-interval processes is introduced, three domain structures are designed based on the key process concept, and VNS is combined to perform local search on the existing scheduling scheme, so that the local search capability of the algorithm is further improved. The method is suitable for workshop scheduling instances of different scales, and compared with existing algorithms, a lower maximum completion time can be achieved on benchmark examples and actual cases.

Description

Technical Field

[0001] This invention belongs to the field of intelligent manufacturing and production scheduling technology, specifically relating to a flexible workshop intelligent scheduling method and system based on Q-Learning enhanced hierarchical sparrow search. Background Technology

[0002] With the arrival of Industry 5.0, the manufacturing industry is gradually shifting away from the traditional mass production model and towards a production model characterized by diverse varieties, small batches, and personalized customization. This transformation effectively meets the diversified and personalized demands of the market, but it also brings a series of challenges to manufacturing enterprises. These challenges include a significant increase in the complexity of the production process, a decrease in the utilization rate of production equipment, and increased difficulty in production scheduling. These problems directly restrict the improvement of enterprise production efficiency and the control of production costs. Against this backdrop, production scheduling, as a core link in optimizing manufacturing processes, improving production efficiency, and reducing operating costs, is becoming increasingly strategically important.

[0003] The Job-Shop Scheduling Problem (JSP) is a classic core problem in the field of production scheduling. In a typical JSP problem, each job needs to be processed through multiple ordered operations, and each operation can only be completed on a single designated machine. Its core optimization objective is to minimize the maximum completion time (Makespan) by rationally planning the processing sequence of each job, thereby improving production efficiency. However, in actual production scenarios, most operations can often be completed by multiple different machines. The traditional JSP's constraint of strictly limiting each operation to a single machine is no longer suitable for the flexible production needs of modern manufacturing enterprises and cannot flexibly respond to dynamic changes in the production process. Therefore, the Flexible Job-Shop Scheduling Problem (FJSP) has emerged.

[0004] As an extension of JSP, FJSP breaks the single correspondence between "process and machine," allowing each workpiece to select multiple alternative machines for each process. This significantly improves the flexibility and adaptability of production scheduling, making the scheduling model more aligned with the complex needs of actual production environments. Compared to traditional JSP, FJSP's scheduling optimization task is more complex, requiring not only the rational determination of the processing sequence for each workpiece and process but also the selection of the optimal processing machine for each process, achieving dual optimization of "sequence planning + machine selection." In-depth research and efficient solutions to FJSP problems can help manufacturing enterprises further improve production efficiency, reduce production costs, and shorten production cycles, thereby enhancing their core competitiveness. This has significant engineering application value and practical guiding significance.

[0005] Existing research confirms that the Free-Journal Problem (FJSP) is an NP-hard problem, meaning that its difficulty increases exponentially with the problem size (number of jobs, processes, and machines). Currently, methods for solving the FJSP problem are mainly divided into two categories: exact methods and swarm intelligence algorithms. Exact methods can theoretically obtain the globally optimal solution, but due to computational complexity limitations, they often cannot obtain an effective solution within a finite time when facing large-scale real-world production scheduling problems, making it difficult to meet the real-time scheduling requirements of actual production. Therefore, most current research focuses on the application of swarm intelligence algorithms, such as genetic algorithms, gray wolf algorithms, and artificial bee colony algorithms, using heuristic search strategies to strive for high-quality approximate optimal solutions within a finite computation time to adapt to the needs of real-world production scenarios.

[0006] However, existing FJSP solution schemes based on swarm intelligence algorithms still have significant limitations and cannot fully meet the scheduling requirements of modern flexible manufacturing. First, the performance of traditional swarm intelligence algorithms is highly sensitive to their parameter settings. Different FJSP problem instances (such as differences in the number of workpieces, process complexity, and machine configuration) often require parameter readjustment to achieve the desired solution. This process not only increases the operational complexity of the system but also significantly increases the cost of parameter tuning, reducing the ease of practical application of the algorithm. Second, during the iterative search process, as the number of iterations increases, the population diversity gradually decreases, causing the algorithm to easily get trapped in local optima and exhibit premature convergence. This, in turn, affects the quality of the final scheduling solution, making it impossible to effectively shorten the maximum completion time and fully realize the production potential of the flexible workshop. Summary of the Invention

[0007] The purpose of this invention is to overcome the aforementioned problems existing in the prior art and to provide a flexible workshop intelligent scheduling method and system based on Q-Learning enhanced hierarchical sparrow search, so as to improve the performance of the algorithm and reduce the maximum completion time.

[0008] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution: This invention provides a flexible intelligent workshop scheduling method based on Q-Learning enhanced hierarchical sparrow search, comprising the following steps: 1) Scheduling scheme encoding and initialization: The flexible job shop scheduling scheme is represented by a dual-vector (MS, OS) encoding, and the initial population is generated by combining the GLR initialization strategy that takes into account machine load and the Tent chaotic mapping. 2) Hierarchical sparrow search and Q-learning co-optimization: The population is hierarchically divided according to fitness, and individuals at lower levels learn from individuals at higher levels. Q-learning is used to dynamically adjust the number of population levels to balance global exploration and local development. 3) Variable neighborhood search for critical processes: Three neighborhood structures are designed for the critical path to enhance local search capabilities and optimize scheduling schemes.

[0009] Furthermore, the dual-vector encoding rule in step 1) is as follows: Machine selection vector MS: It corresponds one-to-one with the workpiece process, and the coded value represents the sequence number of the candidate machine for the process selection; Process sequencing vector OS: consists of repeatable workpiece numbers, and the order in which the numbers appear is the processing order of the process; A bidirectional mapping and transformation rule between continuous individual vectors and discrete scheduling schemes is established.

[0010] Furthermore, the population initialization method in step 1) is as follows: The MS part adopts a GLR hybrid initialization strategy, which integrates three machine allocation methods: global selection, local selection, and random selection. The OS part uses Tent chaotic sequence initialization to improve the uniformity and quality of the initial solution distribution.

[0011] Furthermore, the Q-learning regulation rule in step 2) is: Both states and actions are defined as population hierarchy numbers, and candidate values ​​are a preset set; An ε-greedy strategy is used to select the level, and the reward value is calculated based on the average fitness of the population and the optimal fitness. The Q-value table is iteratively updated using the Q-learning update formula to dynamically adjust the population hierarchy.

[0012] Furthermore, the hierarchical position update rule in step 2) is: The top-level individual executes the sparrow search algorithm's discoverer position update rules; Intermediate-level individuals select higher-level individuals to learn from through a dual tournament, update according to the improved joiner rules, and perform MS and OS learning operations; The bottom-level individuals are updated using a genetic mutation operator to avoid local optima; Randomly select individual vigilants and execute the danger avoidance update rules.

[0013] Furthermore, the three neighborhood structures in step 3) are: Key process machine replacement neighborhood: Randomly replace the optional processing machines for key processes; Key process neighborhood swapping: swapping the process positions of key processes on different workpieces; Reverse Neighborhood of Critical Processes: Reverse the order of the subset of critical processes; The top-level individual enforces a variable neighborhood search, while the remaining individuals perform the search with a preset probability.

[0014] Furthermore, the overall solution process is as follows: initialize parameters and instances → generate initial population → calculate fitness → Q-learning determines the number of levels and levels → update positions level by level → variable neighborhood search → update Q-value table → iterate to the maximum number of times and output the optimal scheduling scheme.

[0015] This invention also provides a flexible intelligent workshop scheduling system based on Q-Learning enhanced hierarchical sparrow search, used to implement the above-mentioned scheduling method, including: The encoding initialization module is used to perform the encoding and initialization operations in step 1). The hierarchical optimization module is used to perform the hierarchical sparrow search and Q-learning co-optimization operation in step 2); The neighborhood search module is used to perform the variable neighborhood search operation, a key step in step 3). The scheduling output module is used to iteratively calculate and output the optimal flexible workshop scheduling scheme.

[0016] In this invention, the algorithm for solving the flexible job shop scheduling problem is as follows: (1) Initialization module, used to construct an initial scheduling scheme population. Machine sequence generation part, used to select processing machines for each process considering the current load of each candidate machine, in order to generate machine sequence. Process sequence generation unit, used to arrange processes based on Tent chaotic mapping to generate process sequence. Among them, machine sequence and process sequence together constitute a double sequence encoded individual used to represent scheduling scheme.

[0017] (2) Hierarchical search module, used to perform a global search of the scheduling scheme population based on a multi-level sparrow search algorithm. The scheduling scheme population is sorted and divided into multiple levels using Q-learning based on individual fitness. Lower-level scheduling schemes learn from and update higher-level scheduling schemes to improve search capabilities. Different levels correspond to different search strategies and update intensities.

[0018] (3) Local optimization module, used to perform local search optimization on the scheduling scheme obtained by hierarchical search, identifies the set of key processes consisting of processes with no idle intervals from the current scheduling scheme, and uses a variable neighborhood search strategy to switch between different neighborhood structures to perform local search on the scheduling scheme. Among them, the neighborhood structure includes at least the key process machine replacement neighborhood, the key process exchange neighborhood, and the key process reverse order neighborhood.

[0019] The beneficial effects of this invention are: 1. Two sequences, MS and OS, are generated using the GLR strategy and Tent chaotic mapping respectively to obtain a high-quality initial scheduling scheme population. Specifically, a hierarchical sparrow search algorithm (QL-HSSA) based on Q-learning and dynamically adjusting the number of levels is designed. By updating the population hierarchically, the diversity of the population is significantly improved, the risk of the algorithm getting trapped in local optima is reduced, and the overall optimization performance and convergence speed are enhanced. The concept of a critical process consisting of no-interval processes is introduced, and three neighborhood structures are designed based on this. Combined with VNS, the algorithm performs local search on existing scheduling schemes, further improving its local search capability.

[0020] 2. The method of the present invention is applicable to workshop scheduling instances of different sizes. Compared with existing algorithms, it can achieve a lower maximum completion time in benchmark cases and actual cases.

[0021] Of course, any product implementing this invention does not necessarily need to achieve all of the above advantages at the same time. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is the bi-vector encoding diagram involved in the algorithm of this invention; Figure 2 The Gantt chart involved in the algorithm of this invention; Figure 3 This is a diagram showing the relationship between the workpiece number and the position index of the present invention; Figure 4 This is a diagram showing the conversion between individual vectors and scheduling schemes in this invention; Figure 5 This is a diagram illustrating the global selection process of the present invention; Figure 6 This is a diagram illustrating the MS and OS learning operations of this invention; Figure 7 This is a diagram illustrating the MS and OS mutation operations of the present invention; Figure 8 This is a diagram showing the three neighborhood structures of the present invention; Figure 9 This is a flowchart of the QL-HSSA of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] like Figures 1-9 As shown, this embodiment provides a flexible workshop intelligent scheduling method based on Q-Learning enhanced hierarchical sparrow search, including the following steps: Scheduling scheme coding and initialization The FJSP problem comprises two subproblems: determining the machine for each job's operations (machine selection) and determining the order in which the operations are performed (operation sequencing). Therefore, a scheduling scheme consists of two sequences to represent these two problems: the machine sequence (…). MS ) and process sequence ( OS ).

[0026] MS The codes correspond to the processes of each workpiece in ascending order of workpiece number. The numbers in the codes indicate which machine to select from all available processing machines for that process. OS It contains repeatable workpiece numbers to be processed; the order in which identical numbers appear corresponds to the process sequence of the workpiece. Assume... MS ={1,2,1,1,2,2} MS The first element represents o 11 choose M 1. Processing. Assume... OS ={2,3,1,1,2,3} represents the processing sequence of the workpiece as { o 21 , o 31 , o 11 , o 12 , o 22 , o 32}

[0027] Given that the FJSP problem is a discrete combinatorial optimization problem, its scheduling scheme consists of discrete variables of machine selection and process sequence. However, the QL-HSSA algorithm is a swarm intelligence algorithm used to solve problems in continuous space, where individuals are typically represented by continuous real numbers. Therefore, QL-HSSA cannot be directly used to solve the FJSP problem; a mapping relationship between the continuous individual vector and the discrete scheduling scheme must be established.

[0028] (1) Scheduling scheme transforms individual vectors for MS The conversion is performed according to the following formula (1): (1) in, m This indicates the total number of machines used for processing. j Indicates the index of the machine sequence. s ( j ) indicates an index j The number of available machines for the corresponding process. u ( j ) indicates an index in the scheduling scheme j The corresponding value, x ( j ) represents the value of the sparrow's individual vector.

[0029] for OS First, some random numbers are generated. Then, according to the ranked order value (ROV) rule, each number is assigned an ROV value. Next, the numbers to be converted are arranged from left to right according to their process number positions. OS Assign a corresponding position index, and finally assign a random number with the same ROV value to each process according to the position index.

[0030] (2) Individual Vector Transformation Scheduling Scheme for MS The transformation is performed using the inverse formula of formula (1): (2) for OS The ROV value is obtained by sorting each value of the individual vector in ascending order, and then the ROV value is converted into the corresponding process code in the process number according to the position index.

[0031] During the population initialization phase, for MS In this part, a hybrid initialization strategy using GLR is employed, which combines global selection, local selection, and random selection. The global selection strategy initializes a machine load array before machine allocation, then randomly selects a workpiece and assigns available processing machines to each of its operations sequentially. During allocation, the machine with the lightest load is selected based on the load after allocation, and the processing time for that operation is added to the machine load array, and so on for each operation. The local selection strategy is similar to the global selection strategy, but the difference is that after allocating machines for all operations of a workpiece, all machine loads are reset, and machine allocation is performed according to the workpiece number order. The random selection strategy randomly selects one machine from the available processing machines for each operation. OS In this part, the process sequence is initialized using a Tent chaotic sequence to improve the quality and distribution of the initial solution, thereby improving the convergence speed and global search capability of the algorithm.

[0032] Hierarchical Sparrow Search and Q-learning Collaborative Optimization In QL-HSSA, population size N They will be divided into equal groups based on fitness. L Each level. L top Represents the highest level, L btm It represents the lowest level. L top The individual in the middle is mainly responsible for guiding the global search direction, while the other individuals mainly follow the higher-level individuals in the search.

[0033] Define State and Action as the number of levels, which are included in a pre-determined set of candidate values ​​for the number of levels. S ={ l 0, l 1,…, l n The number of levels is updated after each iteration. An appropriate number of levels is chosen based on environmental rewards to better search the solution space.

[0034] Initialization of each state Q All values ​​are set to 0, and the selection of Action is based on... Q Value table uses ε - A greedy strategy is used to select the number of levels. Based on probability. ε Randomly select a level number with probability 1− ε Select Current Q The number of levels with the highest value.

[0035] After selecting an action, the algorithm executes the action and receives a reward value. The reward is based on the average fitness and the best fitness of the population, and the reward function is as follows: (3) in, Indicates the first t The average fitness of the population in each iteration This represents the fitness of the optimal individual. ω 1 and ω 2 is the weighting coefficient, which satisfies ω 1+ ω 2=1, considering that only the optimal scheduling scheme is needed in the end, therefore, during allocation, a minimum allocation will be given. ω 2. Larger value.

[0036] After receiving a reward, the update formula is used to... Q The value table is updated using the following formula: (4) in, α For learning rate, γ As a discount factor, s t and a t These represent the current state and the action, respectively.

[0037] Through this mechanism, the algorithm can dynamically learn the optimal hierarchical structure based on environmental feedback during the search process, thereby improving search capability and convergence speed.

[0038] In the QL-HSSA algorithm, the discoverer-joiner-banner behavior pattern of the original sparrow search algorithm is retained, and individuals at different levels undertake different search tasks to achieve a reasonable division of labor between global exploration and local development within the population.

[0039] (1) Top-level position update For those at the top L top These individuals, possessing the best fitness in the current population, are responsible for guiding the population's search direction. They continue to follow the behavior pattern of the discoverer in the original sparrow search algorithm, updating their positions according to formula (5).

[0040] (5) in, t Represents the current iteration number. Iteration t The era i The first sparrow j Dimension value, It is a random number between (0, 1). Represents the maximum number of iterations. These represent the warning value and the safe value, respectively. It is a random number that follows a normal distribution. It is a vector with the same dimensions as an individual sparrow, and all elements are 1.

[0041] (2) Other level position updates For those in non- L btmOther individuals at each level primarily follow higher-level individuals for position updates, exhibiting the joiner behavior pattern of the original sparrow search algorithm. Here, the joiner behavior pattern is improved to reduce reliance on the best individual. During updates, two individuals above the current level are selected as learning examples for position updates. Considering that random selection limits convergence efficiency, a dual tournament selection mechanism is used. First, a hierarchical binary tournament is conducted to select higher-level individuals. Then, within this level, a second binary tournament is held to select individuals, ultimately choosing one individual as the learning example for the current individual's position update. This operation is performed twice to select two learning examples. This mechanism effectively increases the probability of selecting high-level, excellent individuals, thereby improving the guidance of excellent solutions to the entire population and enhancing the algorithm's convergence. The update formula is: (6) in, , This indicates individuals selected using the dual tournament algorithm. r 1. r 2 and r 3 is a random number between (0, 1).

[0042] Furthermore, considering that higher-level individuals have better scheduling schemes and lower maximum completion times, a further scheduling based on [previous method / mechanism] is introduced after the position update. MS and OS The learning process aims to obtain a better scheduling scheme based on the existing solution. The specific operations are as follows: Step 1: MS Learning operation, randomly generate two different integers to determine in MS The sub-intervals of the learning process are selected, and the best learning examples within that interval are extracted. MS The subsequence is directly replaced to the corresponding position of the current individual, while the machine selection for the remaining processes remains unchanged.

[0043] Step 2: OS The learning process involves randomly selecting a subset of workpiece operations from the best learning examples and inserting these operations sequentially into the corresponding positions of the current individual workpiece, according to their order in the learning examples. The operations of the remaining workpieces are then filled into the remaining empty positions, maintaining their original relative order.

[0044] In the original SSA algorithm, entrants with lower fitness will fly to other areas to forage due to hunger. Here, considering that the original update formula updates the individual vector, which might not change much after being transformed into a scheduling scheme, thus failing to effectively escape the current area, the mutation operator of a genetic algorithm is used to replace the original update formula. For those in... L btm individuals, MS Partially, a half-mutation is used, where half of the randomized processes are replaced with other machines in an optional processing set. OS Partially, neighbor mutation is used, selecting three different processes, shuffling their original order, and inserting them into new positions.

[0045] (3) Guardian location update The behavior pattern of the watchers in the original sparrow search algorithm is retained. For the watchers, the group consists of random individuals from all levels of the population, with a quantity of [missing information]. SD %, when danger is detected, it will abandon the search for food and move to a new location. Update according to formula (7): (7) in, This represents the globally optimal position after iteration t. It is a step size control parameter with a mean of 0 and a variance of 1. , and Let represent the current individual sparrow's fitness, the global best fitness, and the global worst fitness, respectively. It is a very small constant, the purpose of which is to... To prevent the denominator from being 0.

[0046] Variable Neighborhood Search for Key Processes To further enhance the local search capability of QL-HSSA, a Variable Neighborhood Search (VNS) strategy is introduced. VNS effectively improves local search efficiency by continuously switching between different neighborhood structures, thereby preventing the algorithm from getting trapped in local optima.

[0047] In the FJSP problem, the critical path refers to the longest sequence of operations that determines the total completion time of the entire schedule. It consists of a series of seamless operations, called critical operations. Moving critical operations can effectively reduce the maximum completion time. Based on this, the following three neighborhood operations were designed for both the MS and OS: (1) Replace the neighboring area of ​​the critical process machine. Randomly select a process from the set of critical processes. If there are multiple optional processing machines for the process, randomly replace it with another processing machine to generate a new neighboring solution. This operation aims to change the processing machine of the critical operation, thereby changing the critical path and reducing the completion time.

[0048] (2) Exchange neighborhood of critical operations: Randomly select two different workpiece operations in the critical operation set and exchange their positions to generate a new neighborhood solution. This operation aims to change the processing order of critical operations, thereby changing the critical path and reducing the completion time.

[0049] (3) Reverse neighborhood of critical process: Randomly select two processes from the set of critical processes and reverse the process sequence between them to generate a new neighborhood solution. This operation aims to change the processing order of critical operations, thereby changing the critical path and reducing the completion time.

[0050] In application, for those in L top Individuals at each level will perform a VNS operation once, while other individuals will... r vns The probability of performing a VNS operation is determined, and if a better solution is found, the original solution is replaced.

[0051] Overall solution process The specific steps of QL-HSSA are as follows: Step 1: Initialize the parameters required by the algorithm and the input FJSP instance. Step 2: Generate an initial population using the GLR strategy and the Tent chaotic map. Step 3: Calculate the fitness of each individual. Step 4: Use... ε - The greedy strategy determines the number of levels L to be divided, and the population is divided according to fitness. L Step 5: Update the position of individuals at different levels. Step 6: Perform VNS operations on all individuals with a certain probability. Step 7: Calculate the reward value and update the Q-value table. Step 8: Repeat steps 3 to 7 until the maximum number of iterations is reached.

[0052] To verify the effectiveness of the method in this embodiment, the inventors implemented the proposed algorithm through computer simulation. The dataset used is briefly described below.

[0053] The properties of the workpiece of this invention are shown in Table 1: Table 1

[0054] In this experiment, the inventors used the benchmark simulations MK01 to MK10 designed by Brandimarte and a real-world case. The MK01 to MK10 simulations include instances of different scales, from 10 workpieces and 6 machines to 20 workpieces and 15 machines, covering small, medium, and large-scale production simulations. The real-world case comes from a simulation based on the actual needs of a factory.

[0055] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A flexible intelligent workshop scheduling method based on Q-Learning enhanced hierarchical sparrow search, characterized in that, Includes the following steps: 1) Scheduling scheme encoding and initialization: The flexible job shop scheduling scheme is represented by a dual-vector (MS, OS) encoding, and the initial population is generated by combining the GLR initialization strategy that takes into account machine load and the Tent chaotic mapping. 2) Hierarchical sparrow search and Q-learning co-optimization: The population is hierarchically divided according to fitness, and individuals at lower levels learn from individuals at higher levels. Q-learning is used to dynamically adjust the number of population levels to balance global exploration and local development. 3) Variable neighborhood search for critical processes: Three neighborhood structures are designed for the critical path to enhance local search capabilities and optimize scheduling schemes.

2. The scheduling method according to claim 1, characterized in that, The dual-vector encoding rule in step 1) is as follows: Machine selection vector MS: It corresponds one-to-one with the workpiece process, and the coded value represents the sequence number of the candidate machine for the process selection; Process sequencing vector OS: consists of repeatable workpiece numbers, and the order in which the numbers appear is the processing order of the process; A bidirectional mapping and transformation rule between continuous individual vectors and discrete scheduling schemes is established.

3. The scheduling method according to claim 1, characterized in that, The population initialization method in step 1) is as follows: The MS part adopts a GLR hybrid initialization strategy, which integrates three machine allocation methods: global selection, local selection, and random selection. The OS part uses Tent chaotic sequence initialization to improve the uniformity and quality of the initial solution distribution.

4. The scheduling method according to claim 1, characterized in that, The Q-learning regulation rule in step 2) is: Both states and actions are defined as population hierarchy numbers, and candidate values ​​are a preset set; An ε-greedy strategy is used to select the level, and the reward value is calculated based on the average fitness of the population and the optimal fitness. The Q-value table is iteratively updated using the Q-learning update formula to dynamically adjust the population hierarchy.

5. The scheduling method according to claim 1, characterized in that, The hierarchical position update rule in step 2) is: The top-level individual executes the sparrow search algorithm's discoverer position update rules; Intermediate-level individuals select higher-level individuals to learn from through a dual tournament, update according to the improved joiner rules, and perform MS and OS learning operations; The bottom-level individuals are updated using a genetic mutation operator to avoid local optima; Randomly select individual vigilants and execute the danger avoidance update rules.

6. The scheduling method according to claim 1, characterized in that, The three neighborhood structures in step 3) are: Key process machine replacement neighborhood: Randomly replace the optional processing machines for key processes; Key process neighborhood swapping: swapping the process positions of key processes on different workpieces; Reverse Neighborhood of Critical Processes: Reverse the order of the subset of critical processes; The top-level individual enforces a variable neighborhood search, while the remaining individuals perform the search with a preset probability.

7. The scheduling method according to claim 1, characterized in that, The overall solution process is as follows: initialize parameters and instances → generate initial population → calculate fitness → Q-learning determines the number of levels and levels → update positions level by level → variable neighborhood search → update Q-value table → iterate to the maximum number of times and output the optimal scheduling scheme.

8. A flexible workshop intelligent scheduling system based on Q-Learning enhanced hierarchical sparrow search, used to implement the scheduling method described in any one of claims 1-7, characterized in that, include: The encoding initialization module is used to perform the encoding and initialization operations in step 1). The hierarchical optimization module is used to perform the hierarchical sparrow search and Q-learning co-optimization operation in step 2); The neighborhood search module is used to perform the variable neighborhood search operation, a key step in step 3). The scheduling output module is used to iteratively calculate and output the optimal flexible workshop scheduling scheme.

Images