An energy-saving fuzzy cascade scheduling method and system for regional clustered collaborative production

Abstract

This invention relates to the field of intelligent manufacturing technology, specifically to an energy-saving fuzzy cascade scheduling method and system for regional clustered collaborative production. This method addresses complex issues in the regional collaborative transformation of the manufacturing industry, including resource allocation in heterogeneous factories within the supply chain cascade scheduling, multi-stage time accumulation effects, and uncertainty interference. The method employs an ensemble learning-assisted two-layer collaborative optimization framework. It uses a five-tuple interval fuzzy quantification of the uncertainty in processing, transportation, and assembly times. An initial Q-value matrix is ​​generated through a pre-training layer, and an adaptive operator selection is achieved through a dynamic decision layer. Multiple subgroups perform local search, destructive recombination, and genetic operations, and a decoupled crossover strategy expands the solution space. This effectively defines a problem model for the energy-saving second-class fuzzy distributed flow shop and multi-flexible operation shop cascade scheduling. It supports three-segment coding and full-process energy consumption calculation, significantly improving the overall scheduling efficiency and energy efficiency balance capability of regionally clustered industries.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, specifically to an energy-saving fuzzy cascade scheduling method and system for regional clustered collaborative production. Background Technology

[0002] As global manufacturing gradually shifts towards regionalization and collaboration, supply chain production systems face the dual challenges of multi-factory collaborative scheduling and dynamic uncertainty. In complex production scenarios, how to coordinate resource allocation, job sequencing, and fuzzy time constraints among multi-stage factories while simultaneously optimizing energy efficiency has become a core issue that urgently needs to be addressed. However, existing technologies still have the following shortcomings: Limitations of scheduling models: Traditional scheduling models mainly focus on single factories or homogeneous environments, making it difficult to adapt to the complex needs of heterogeneous collaborative production across multiple factories. Especially in regionally clustered industries, upstream and downstream enterprises need to achieve cascaded collaboration between distributed assembly lines and flexible work workshops. Existing models have significant deficiencies in handling dynamic uncertainty and multi-objective collaborative optimization, failing to meet actual production needs; Efficiency bottlenecks of algorithmic frameworks: Existing algorithmic frameworks, such as genetic algorithms and destructive recombination algorithms, heavily rely on fixed search strategies, leading to problems such as strong blindness in early searches and insufficient global convergence ability. The challenge of balancing energy efficiency and response speed: In cascaded scheduling scenarios, maximum completion time directly impacts the supply chain's responsiveness, while total energy consumption is crucial to a company's sustainable development goals. Existing methods often employ single-objective optimization or simple weighted strategies, failing to effectively balance efficiency and energy consumption dynamically, resulting in low resource utilization and difficulty in achieving green production goals. Insufficient uncertainty modeling: In real-world production environments, processing, transportation, and assembly times are significantly fuzzy due to factors such as equipment status and order fluctuations. Traditional scheduling models often employ deterministic or type-one fuzzy systems, making it difficult to accurately describe the distribution characteristics of complex uncertainties. In recent years, regional clustered production models have provided new ideas for multi-factory collaboration by integrating the synergistic advantages of distributed assembly lines and flexible work workshops. However, such models need to address challenges such as cross-factory operation allocation, process connection under fuzzy time constraints, and dynamic energy consumption optimization. While existing research attempts to integrate distributed assembly lines and assembly scheduling, significant shortcomings remain in dynamic decision-making, multi-objective collaboration, and uncertainty handling. Summary of the Invention

[0003] This invention provides an energy-saving fuzzy cascade scheduling method and system for regional clustered collaborative production, in order to solve the problems mentioned above.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] An energy-saving fuzzy cascade scheduling system for regional clustered collaborative production is characterized by including a modeling module, an encoding module, an optimization module, and an output module.

[0006] The modeling module is used to construct a mathematical model for the energy-saving type II fuzzy distributed flow shop and multi-flexible work shop cascade scheduling problem ET2FDFS-MFJSCSP.

[0007] The encoding module is used to generate three-segment codes: workpiece sorting (JS), machine selection (MS), and process sorting (OS).

[0008] The optimization module is used to execute the LBCOF algorithm, including population initialization, subpopulation partitioning, Q-learning decision-making, and decoupling crossover.

[0009] The output module is used to output the Pareto optimal solution set and its corresponding scheduling scheme.

[0010] An energy-saving fuzzy cascade scheduling method for regional clustered collaborative production includes the following process:

[0011] Step 1: Establish a mathematical model for the energy-saving type II fuzzy distributed flow shop and multiple flexible work shop cascade scheduling problem ET2FDFS-MFJSCSP. Set minimizing the maximum completion time and total energy consumption as the dual objectives. The mathematical model of the energy-saving type II fuzzy distributed flow shop and multiple flexible work shop cascade scheduling problem ET2FDFS-MFJSCSP includes fuzzy time and energy consumption models for the processing stage, transportation stage and assembly stage. The processing time, transportation time and assembly time are all represented and calculated using the five-tuple interval type II fuzzy number IT2F.

[0012] The IT2F interval fuzzy number set of the quintuple uses a quintuple. It means that among them For two types of fuzzy sets The center of mass, and Used for constraints The upper boundary of the territory, and Used for constraints The lower membership boundary, upper membership boundary, and lower membership boundary determine the range of the second type of fuzzy set.

[0013] Step 2: Design a three-segment coding method, including workpiece sorting (JS), machine selection (MS), and process sorting (OS), and design corresponding decoding strategies. The decoding strategies include calculating the workpiece completion time based on the workpiece sorting (JS) code, determining the transportation time based on product relationships and calculating the workpiece arrival time at the assembly plant, determining the earliest start time for product assembly, and allocating processes to machines based on the process sorting (OS) and machine selection (MS) codes. An activity scheduling strategy is used to generate a scheduling scheme.

[0014] Step 3: Generate an initial population using a hybrid initialization strategy and divide it into multiple subpopulations. The hybrid initialization strategy includes: the workpiece sorting JS code adopts a product set rule or a random rule; the machine selection MS code adopts a minimum time rule, a minimum load rule or a random rule; the process sorting OS code adopts a first-come-first-served rule, a machine greedy rule or a random rule; and the initial population is generated by combining five rules.

[0015] Step 4: Construct a learning-assisted two-layer co-evolutionary optimization framework LBCOF, including a pre-training layer and a dynamic decision layer.

[0016] Step 5: In the pre-training layer of Step 4, the three subpopulations perform three search operations: local search operation, disruptive recombination operation, and genetic operation, and record the success rate of each operation in the Q table.

[0017] Step 6: In the dynamic decision-making layer of step 4, the search operation is adaptively selected based on the Q-table and ε-greedy learning auxiliary strategy, and a decoupling crossover strategy is executed on the parent individuals to enhance search diversity.

[0018] Step 7: Finally, update the population through environmental selection and non-dominated sorting, and output the Pareto optimal solution set.

[0019] Furthermore, the three search operations in step 5 include local search operations: based on the variable neighborhood descent VND strategy, including four neighborhood structures: js_out_insert, ms_select, os_swap, and js_in_swap; destruction and recombination operations: small-scale destruction and recombination of workpieces, machines, and processes in key plants; and genetic operations: using POX crossover and uniform crossover, combined with mutation operations.

[0020] Furthermore, the Q-table and ε-greedy learning assistance strategy in step 6 include: the state space consists of the number of evaluations and successes of each search operation; the action space consists of three types of search operations; the reward function is based on the difference between the current success rate and the historical average success rate; and a hierarchical Q-learning mechanism is adopted, with the pre-training layer learning offline and the decision layer dynamically selecting operations online.

[0021] Furthermore, the decoupling and crossover strategy in step 6 is as follows: the processing stage coding workpiece sorting JS of the parent individual is separated and recombined with the assembly stage coding machine selection MS and process sorting OS to generate the child individual.

[0022] The present invention has the following beneficial effects:

[0023] This invention defines for the first time the mathematical model of the energy-saving type II fuzzy distributed flow shop and multiple flexible job shop cascade scheduling problem ET2FDFS-MFJSCSP, and proposes a calculation criterion with Cmax and TEC as dual objectives, providing a systematic theoretical framework for multi-factory collaborative optimization. Considering the characteristics of the mathematical model of the energy-saving type II fuzzy distributed flow shop and multiple flexible job shop cascade scheduling problem ET2FDFS-MFJSCSP, three differentiated search strategies—local search, destructive recombination, and genetic operation—are designed to improve solution quality, enhance global exploration, and maintain population diversity, respectively, thereby achieving performance improvement. Balancing efficiency and performance, a learning-assisted search strategy is introduced. Through a hierarchical training mechanism, adaptive adjustment of operator selection is achieved, significantly reducing the impact of strategy randomness on search efficiency and improving the stability and convergence speed of the algorithm. A decoupling crossover strategy is proposed to separate the encoding and reorganization processes of the processing and assembly stages, effectively expanding the search range of the solution space and enhancing the algorithm's adaptability to complex scenarios. The overall system logic is clear and concise, easy to implement and expand, and can adapt to most dynamic scheduling scenarios in the field of intelligent manufacturing. In particular, it exhibits good robustness and scalability in multi-factory heterogeneous environments, meeting actual production needs. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the mathematical model of the energy-saving type II fuzzy distributed flow workshop and multi-flexible operation workshop cascade scheduling problem in this invention, ET2FDFS-MFJSCSP.

[0025] Figure 2 This is the three-segment encoding method in this invention.

[0026] Figure 3 This is the two-layer collaborative evolutionary architecture in this invention.

[0027] Figure 4 This is the algorithm flowchart in this invention.

[0028] Figure 5 This is an embodiment of the present invention.

[0029] Figure 6 This is a box plot of four algorithms under the HV index in this invention.

[0030] Figure 7 This is a box plot of four algorithms under the IGD index in this invention.

[0031] Figure 8 This is a comparison diagram of the Pareto front under S06 conditions.

[0032] Figure 9 This is a comparison diagram of the Pareto front under S14 conditions.

[0033] Figure 10 This is a comparison diagram of the Pareto front under L01 conditions.

[0034] Figure 11 This is a comparison diagram of the Pareto front under L12 conditions. Detailed Implementation

[0035] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0036] The energy-saving fuzzy cascade scheduling method and system for regional clustered collaborative production provided by this invention are based on the learning-assisted two-layer co-evolutionary optimization framework LBCOF and are implemented through the following steps:

[0037] The first step is to establish a mathematical model for the energy-saving type II fuzzy distributed flow workshop and multi-flexible operation workshop cascade scheduling problem ET2FDFS-MFJSCSP.

[0038] Table 1. Notes on the Mathematical Model

[0039]

[0040]

[0041] The goal of the mathematical model for the energy-saving type II fuzzy distributed flow shop and multi-flexible work shop cascade scheduling problem ET2FDFS-MFJSCSP is to minimize the maximum completion time Cmax and the total energy consumption TEC.

[0042]

[0043]

[0044]

[0045]

[0046] The second step is the encoding and decoding design.

[0047] In the mathematical model of the ET2FDFS-MFJSCSP problem of cascade scheduling of energy-saving type II fuzzy distributed flow shop and multiple flexible job shop, the production stage needs to determine the factory allocation of workpieces and their processing sequence within each factory, while the assembly stage needs to decide on the machine selection and process sequencing of each assembly factory. For example... Figure 2 As shown, the solution for the distributed flow shop in the production stage is encoded by a vector denoted as job ordering (JS), where job sequences between different factories are separated by 0. For the assembly stage, the solution for multiple flexible job shops is represented by two vectors, machine selection (MS) and process ordering (OS). The assembly stage is arranged according to the order of the assembly plants. Within each assembly plant, machine selection (MS) is arranged according to the product and process order of that plant, using the processing machine selected for the current process as the encoding gene; while process ordering (OS) adopts a process-based encoding method, directly using the product number as the encoding gene. Therefore, the overall encoding of the solution of the mathematical model of the energy-saving type II fuzzy distributed flow shop and multiple flexible job shop cascade scheduling problem ET2FDFS-MFJSCSP consists of a triple {JS, MS, OS}, where the triple consists of job ordering (JS), machine selection (MS), and process ordering (OS).

[0048] Since the mathematical model of the energy-saving type II fuzzy distributed flow shop and multi-flexible job shop cascade scheduling problem ET2FDFS-MFJSCSP includes two interrelated stages: production and assembly, the earliest assembleable time for all products is determined by the arrival time of the last workpiece of that product at the assembly plant. Furthermore, because the assembly stage involves multiple flexible job shops, an active scheduling strategy can be adopted to improve scheduling flexibility. The decoding process steps are as follows:

[0049] 1. Based on the workpiece sorting in the job sequence, JS calculates the completion time of each workpiece and records it as... .

[0050] 2. Determine the transportation time based on the product structure relationship of each workpiece, and record it as follows: ; then The time for each workpiece to arrive at the designated assembly plant is calculated using a two-class fuzzy addition method.

[0051] 3. The completion time of the last completed workpiece among all the workpieces required for each product shall be taken as the earliest time when assembly of that product can begin.

[0052] 4. According to the order specified by the process sequencing OS and the earliest assembly start time of each product, the processes of each product are sequentially assigned to the machine selection MS of the corresponding machine. The earliest start time of each process depends on the completion time of the previous process or the earliest arrival time of the workpiece.

[0053] 5. Based on the earliest start time of the process and the current scheduling state of the machine, determine whether the process can be inserted before other processes on the current machine. If the insertion condition is met, perform the insertion operation; otherwise, place it at the end of the machine's scheduling sequence. At this point, the decoding process is complete, and the corresponding activity scheduling scheme is generated.

[0054] The third step is to use the LBCOF algorithm to solve the mathematical model of the energy-saving type II fuzzy distributed flow shop and multiple flexible work shop cascade scheduling problem ET2FDFS-MFJSCSP.

[0055] The flowchart of the proposed LBCOF algorithm is as follows: Figure 4 As shown, the algorithm description is in Algorithm 1, and its termination condition is set to the total number of fitness evaluations TNFEs.

[0056] Algorithm 1

[0057]

[0058] A hybrid initialization rule was used to generate the initial population, and differentiated initialization rules were designed based on the characteristics of each coding segment. The initialization rules for the three coding segments are given below: the workpiece sorting JS uses two rules, namely product set and random; the machine selection MS introduces three rules, namely time optimal, load balancing and random allocation; and the process sorting OS developed three rules, namely first-come-first-served, machine greedy and random.

[0059] The specific combination methods of the basic rules for workpiece sorting (JS), machine selection (MS), and process sorting (OS) are shown in Table 2.

[0060] Table 2. Combination Methods

[0061]

[0062] This strategy generates five complementary initial solution generation strategies using the combination methods shown in Table 2, where IJR, IMR, and IOR represent the initialization rules for the corresponding coding segments. Each strategy independently generates a subpopulation of size PS / 5, and the complete initial population is finally constructed by merging the subpopulations.

[0063] The algorithm employs a two-layer computational resource allocation mechanism, where parameters... Control the resource allocation in the first phase. Define the greedy factor for the second stage of Q-learning. The first stage (steps 2-14) constructs a system containing... The initial population of solutions is determined and evenly divided into groups. Three subpopulations. Each subpopulation has a preset fitness window. The evolutionary process is performed independently within each iteration. After each iteration, an environment selection mechanism based on fast non-dominated sorting and crowding distance metric is executed, simultaneously updating the current number of evaluations (NFEs) and the Q-table. The second stage (steps 15-26) first merges the three subpopulations to construct a new initial population, introducing a decoupling crossover strategy to expand the search range, and employing... The strategy dynamically selects search operators. By merging parent and child generations to perform multi-objective environment selection, an elitist preservation strategy is ultimately employed to extract the Pareto front solution set from the updated population. .

[0064] Step 4: Simulation results and analysis of LBCOF.

[0065] To verify the effectiveness of the LBCOF algorithm in solving the mathematical model of the energy-saving type II fuzzy distributed flow shop and multiple flexible job shop cascade scheduling problem ET2FDFS-MFJSCSP, this study constructed a dataset containing 32 test instances. The number of jobs in the small-scale instances is... The values ​​are {20, 30}, and the product quantity t is {6, 8}; for large-scale instances The values ​​are {80, 100}, and t is {20, 30}. In each instance, the number of processing plants F is {2, 3}, the number of assembly plants Q is {2, 3}, the number of machines m is randomly selected from {6, 7, 8}, and the number of operations o required for each product is randomly selected from {4, 5, 6}. The processing time for the workpiece... and product assembly time All follow a uniform distribution on the interval [10, 50]; transportation time The power follows a uniform distribution on the interval [50, 100]. Processing and assembly power are both 3.0 kWh, transportation power is 2.0 kWh, and idle power is 1.0 kWh. All instances are run independently 10 times, and the algorithm termination condition is set as follows: .

[0066] All experiments were implemented in Python 3.11, running on Windows 11, with a 12th generation Intel® Core™ i7-12700H processor (2.30 GHz) and 332 GB of memory. The performance of the PBIGA algorithm was evaluated using two metrics: hypervolume (HV) and inverse generational distance (IGD) to ensure the scientific validity and reliability of the experimental results.

[0067] In the parameter calibration experiment, the population size was selected. Division ratio With greed factor Three key parameters were analyzed. The value range of each parameter was determined based on preliminary experimental results. Table 2 shows the analysis of variance (ANOVA) results for the IGD index. Statistical analysis shows that, under the HV index, It is a significant influencing factor; while under the IGD index, and All of them have a significant impact. and The p-values ​​are all less than 0.05, indicating that both play an important role in the algorithm's performance. Considering all evaluation results, when the parameter combination is set to... At that time, the LBCOF algorithm showed the best performance.

[0068] Table 3. ANOVA results of LBCOF under IGD index

[0069]

[0070] The LBCOF algorithm was compared with three current mainstream state-of-the-art algorithms—KNSGA-II, KBOA, and LBPEA. The performance results of the four algorithms on HV (hypervolume) and IGD (inverse generational distance) metrics are shown in Table 3. The experimental results show that LBCOF outperforms the comparison algorithms in both convergence performance and the balance of solution set distribution. From the overall trend analysis, LBCOF consistently leads in HV values ​​across all test datasets, indicating that its generated solution set can cover a wider range of high-quality target space regions. This advantage is mainly attributed to the hierarchical collaborative mechanism employed in the algorithm design, which effectively improves the collaborative efficiency between global exploration and local development by dynamically allocating differentiated search strategies.

[0071] Table 4. Average metrics of the four algorithms on 32 datasets.

[0072]

[0073] Figure 6 , Figure 7 Box plots show the distributions of four algorithms based on HV (hypervolume) and IGD (inverse generation distance) metrics. In these box plots, the horizontal line inside the box represents the median, and the hollow square dots represent the mean. Overall, LBCOF demonstrates significant performance advantages and higher stability across both evaluation metrics.

[0074] Under the HV metric, LBCOF's median and mean are significantly higher than the other three algorithms, indicating its stronger optimization capability in terms of solution set coverage and quality. Furthermore, the interquartile ranges of the four algorithms are similar, suggesting they are essentially consistent in terms of solution stability and data dispersion. Regarding the IGD metric, LBCOF's median and mean are significantly lower than the other algorithms, reflecting its superior performance in approximating the true Pareto front. Simultaneously, LBCOF has the smallest interquartile range among the four algorithms, further validating its good convergence capability under fuzzy time constraints and cascaded scheduling mechanisms.

[0075] In contrast, KBOA's median and mean are higher than LBPEA's, which may be due to its fixed policy selection mechanism being less adaptable to complex energy consumption constraints. KNSGA-II, on the other hand, exhibits the highest median and mean, as well as the largest interquartile range, indicating that traditional multi-objective algorithms are less robust in complex problem environments, especially in large-scale datasets where the dispersion of their IGD values ​​is more significant.

[0076] Figure 8-11 The Pareto front distribution shown further illustrates the significant advantages of LBCOF in multi-objective optimization. As can be seen from the figure, the solution set generated by LBCOF exhibits better convergence and distribution uniformity in the dual-objective space. Its solution set is closer to the origin, indicating that the algorithm is closer to the theoretical global optimum in simultaneously minimizing both objective functions. This advantage is not only reflected in the location of the solution set but also in the quality and distribution density of the solutions, further validating the effectiveness and stability of LBCOF in multi-objective scheduling problems.

Claims

1. An energy-saving fuzzy-cascading scheduling system for regional cluster collaborative production, characterized in that, It includes a modeling module, a coding module, an optimization module, and an output module; The modeling module is used to construct a mathematical model for the energy-saving type II fuzzy distributed flow shop and multi-flexible operation shop cascade scheduling problem ET2FDFS-MFJSCSP. The model includes fuzzy time and energy consumption models for the processing stage, transportation stage and assembly stage. The processing time, transportation time and assembly time are all represented and calculated using the five-tuple interval type II fuzzy number IT2F, and the dual objectives are to minimize the maximum completion time and total energy consumption. The encoding module is used to generate a three-segment encoding of workpiece sorting (JS), machine selection (MS), and process sorting (OS). The optimization module is used to execute the LBCOF algorithm. The execution process includes population initialization, subpopulation partitioning, Q-learning decision-making, and decoupling crossover. The LBCOF framework includes a pre-training layer and a dynamic decision-making layer. The three subpopulations in the pre-training layer perform local search, destructive recombination search, and genetic search operations, respectively, and record the success rate of each operation in the Q-table. The dynamic decision-making layer adaptively selects search operations based on the Q-table and the ε-greedy learning auxiliary strategy, and separates and recombines the processing stage code workpiece sorting JS and the assembly stage code machine selection MS and process sorting OS of the parent individual to generate offspring individuals, thereby achieving decoupling crossover. The output module is used to output the Pareto optimal solution set and its corresponding scheduling scheme.

2. An energy-saving fuzzy cascade scheduling method for regional clustered collaborative production, characterized by: The process includes the following steps: Step 1: Establish a mathematical model for the energy-saving type II fuzzy distributed flow shop and multiple flexible work shop cascade scheduling problem ET2FDFS-MFJSCSP. Set minimizing the maximum completion time and total energy consumption as the dual objectives. The mathematical model of the energy-saving type II fuzzy distributed flow shop and multiple flexible work shop cascade scheduling problem ET2FDFS-MFJSCSP includes fuzzy time and energy consumption models for the processing stage, transportation stage and assembly stage. The processing time, transportation time and assembly time are all represented and calculated using the five-tuple interval type II fuzzy number IT2F. Step 2: Design a three-segment coding method, including workpiece sorting (JS), machine selection (MS), and process sorting (OS), and design corresponding decoding strategies. The decoding strategies include calculating the workpiece completion time based on the workpiece sorting (JS) code, determining the transportation time based on product relationships and calculating the workpiece arrival time at the assembly plant, determining the earliest start time for product assembly, and allocating processes to machines based on the process sorting (OS) and machine selection (MS) codes. An active scheduling strategy is used to generate a scheduling scheme. Step 3: Generate an initial population using a hybrid initialization strategy and divide it into multiple subpopulations. The hybrid initialization strategy includes: the workpiece sorting JS code adopts a product set rule or a random rule; the machine selection MS code adopts a minimum time rule, a minimum load rule or a random rule; the process sorting OS code adopts a first-come-first-served rule, a machine greedy rule or a random rule; and the initial population is generated by combining five rules. Step 4: Construct a learning-assisted two-layer co-evolutionary optimization framework LBCOF, including a pre-training layer and a dynamic decision layer; Step 5: In the pre-training layer of Step 4, the three subpopulations perform three search operations respectively: local search operation, disruptive recombination operation and genetic operation, and record the success rate of each operation in the Q table; Step 6: In the dynamic decision-making layer of Step 4, the search operation is adaptively selected based on the Q-table and ε-greedy learning auxiliary strategy, and the decoupling crossover strategy is executed on the parent individual: the processing stage coding workpiece sorting JS of the parent individual is separated and recombined with the assembly stage coding machine selection MS and process sorting OS to generate the offspring individual, thereby completing the decoupling crossover. Step 7: Finally, update the population through environmental selection and non-dominated sorting, and output the Pareto optimal solution set.

3. The energy-saving fuzzy cascade scheduling method for regional clustered collaborative production according to claim 2, characterized in that: The three search operations in step 5 include local search operations: based on the variable neighborhood descent VND strategy, including four neighborhood structures: js_out_insert, ms_select, os_swap, and js_in_swap; destruction and recombination operations: small-scale destruction and recombination of workpieces, machines, and processes in key plants; and genetic operations: using POX crossover and uniform crossover, combined with mutation operations.

4. The energy-saving fuzzy cascade scheduling method for regional clustered collaborative production according to claim 2, characterized in that: The Q-table and ε-greedy learning assistance strategy in step 6 include: the state space consists of the number of evaluations and successes of each search operation; the action space consists of three types of search operations; the reward function is based on the difference between the current success rate and the historical average success rate; and a hierarchical Q-learning mechanism is adopted, with the pre-training layer learning offline and the decision layer dynamically selecting operations online.

Images