Remanufacturing workshop scheduling method based on reinforcement learning and evolutionary algorithm fusion

Abstract

The invention provides a remanufacturing workshop scheduling method based on reinforcement learning and evolutionary algorithm fusion, relates to the technical field of production scheduling optimization, and solves the problem of remanufacturing system scheduling in three stages of disassembly, reprocessing and assembly. A four-dimensional decision model covering process sorting, factory distribution, machine selection and speed gear scheduling is constructed, and an initial population with diversity and high-quality characteristics is generated by adopting a multi-strategy hybrid initialization method. A reinforcement learning decision module based on Q-learning is introduced, an optimal combination is dynamically selected from various crossover operators, mutation operators and neighborhood search strategies, the crossover probability and the mutation probability are dynamically corrected according to the distribution characteristics of the Pareto leading edge, and a dual escape mechanism is integrated to enhance the capability of the algorithm to jump out of local optimum. According to the scheme, the group search advantage of the evolutionary algorithm is exerted, the dynamic optimization of the search process is realized, and an efficient and accurate scheduling solution is provided for a complex remanufacturing system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of production scheduling optimization, in particular to a remanufacturing workshop scheduling method based on reinforcement learning and evolutionary algorithm fusion. BACKGROUND

[0002] At present, in the field of remanufacturing workshop scheduling, the traditional scheduling model mainly targets at the isolated production stage. It fails to coordinate and optimize the three stages of disassembly, distributed reprocessing and assembly, leading to disconnection of the process and limited overall optimization effect. Secondly, in the heterogeneous distributed factory environment, the processing capacity of different factory equipment varies greatly. The traditional method lacks an effective mechanism for cross-factory, machine and speed level coordination decision-making, which hinders the effective allocation of resources. In addition, the traditional evolutionary algorithm for solving such scheduling problems usually iterates the population through pre-defined crossover and mutation. Such search strategy is very single, lacks intelligence and relevance, and is a fixed optimization mode. At the same time, the algorithm parameters are usually fixed and cannot be adaptively adjusted according to the convergence status of the population. When solving complex multi-objective problems, it is easy to fall into local optimum, low solution set quality and insufficient convergence precision, and cannot dynamically adapt to the needs of different stages in the search process. Finally, the existing method often focuses on time optimization and does not consider the energy consumption problem in the production process, especially the dynamic correlation between multi-speed gears of equipment and energy consumption, which is insufficient and difficult to achieve the goal of energy saving and emission reduction while ensuring production efficiency.

[0003] Therefore, it is necessary to provide a remanufacturing workshop scheduling method based on reinforcement learning and evolutionary algorithm fusion to solve the above problems. SUMMARY

[0004] The purpose of the present application is to provide a remanufacturing workshop scheduling method based on reinforcement learning and evolutionary algorithm fusion, which solves the joint scheduling problem of production resources in a distributed heterogeneous environment by solving a three-stage integrated optimization model of disassembly, reprocessing and assembly, and introducing a multi-gear intelligent switching strategy and a Q-learning dynamic decision-making mechanism.

[0005] To achieve the above purpose, the present application provides a remanufacturing workshop scheduling method based on reinforcement learning and evolutionary algorithm fusion, comprising the following steps: S1: Construct a three-stage remanufacturing system model covering disassembly, distributed heterogeneous reprocessing and assembly; S2: Define decision variables including factory selection, machine selection, process ordering and speed gear allocation, establish maximum completion time, total energy consumption optimization objectives and constraint conditions; S3: Use four-dimensional coding to represent the solution of the remanufacturing system model in S1, and use a multi-strategy initialization method to generate an initial population; S4: dynamically adjust the crossover probability and mutation probability according to the search progress of the algorithm using an adaptive method; S5: an intelligent decision-making mechanism based on Q-learning, dynamically selecting the optimal combination of crossover operators, mutation operators and neighborhood search strategies of the NSGA-II algorithm according to the real-time search state; the core decision-making mechanism is based on the following formula: ; wherein is the learning rate, is the discount rate, and the value range is set to (0, 1], represents the current value estimate of selecting action in state , provides immediate performance feedback of the current action; represents the highest expected Q value that can be obtained in the Q value table by performing action in state ; S6: perform the crossover strategy, mutation strategy and local search strategy on the quasi-group in S3; S7: perform fast non-dominated sorting on the population obtained in S6, calculate the crowding degree, and generate a new generation of population based on the elite preservation strategy; S8: determine whether the current generation has reached the maximum number of iterations, if the result is yes, end the iteration and output the optimal scheduling scheme; if the result is no, execute the next step S9; S9: execute the escape mechanism, introduce new random individuals to help the NSGA-II algorithm jump out of the local optimal region, and return to S4 to continue iteration until the maximum number of iterations is reached and the optimal scheduling scheme is output.

[0006] Preferably, the maximum completion time in S2 is represented as: ; wherein represents the maximum assembly completion time of the product; The total energy consumption function is represented as: ; wherein E1 represents the total disassembly energy consumption; E2 represents the total reprocessing energy consumption; E3 represents the total assembly energy consumption; E4 represents the system fixed energy consumption; The constraint condition is represented as: Each operation of the workpiece can only be processed on one machine: ; In the formula, This represents the decision variable, specifically the defective workpiece. j Operation i Is it in the machine? k To perform processing; what does M represent; Each operation can only select one machine of a certain speed class from the machine set: ; In the formula, Choose a decision variable for speed, representing the operation. i Is it based on speed? In the machine k Execute above.

[0007] Preferably, in S3, the solution of the earlier system model is represented by a four-dimensional encoded process sorting vector, factory allocation vector, machine allocation vector, and speed selection vector. The proportion of the initialization strategy in the initialization phase is adjusted to generate the initialization population. The specific allocation strategy is expressed as follows: The shortest processing time priority rule accounts for 40%, prioritizing tasks with shorter processing times and optimizing completion time targets; Intelligent allocation strategies based on factory machine resource allocation account for 40%, prioritizing factories with the most idle machines. Random initialization accounts for 20%, and solutions are generated randomly.

[0008] Preferably, the crossover probability pc and mutation probability pm in S4 are calculated as follows: ; ; ; In the formula, and The factor representing the regulation of crossover and mutation has a value range of (0,1]. N The number of solutions in the current Pareto front; Indicates the first p The objective function at the th ... q The value on each solution; Indicates the first p In the objective function, with the first q The target value of the individual whose solution is closest to the target value; This describes the total dispersion of all solutions on the current Pareto front; This represents the relative proportion between the current solution and the optimal solution; Represents the natural constant.

[0009] Preferably, in S5, by continuously iterating and updating the Q-value table, the Q-learning algorithm gradually learns the value mapping of the optimal operator combination under different search states and establishes the optimal decision strategy; state Assess the convergence and diversity of the current Pareto frontier; Convergence state ΔGD Using generational distance GD As a convergence evaluation metric, GD The Pareto solution set that measures the current population P With reference to Pareto Frontier The average minimum distance between them is calculated using the following formula: ; in, Indicates population P The Middle q Individual to reference set Euclidean distance to the nearest individual; GD The smaller the value, the better the convergence of the population; during the iteration process, the change in the convergence state is defined as: ;when When, it indicates that the population convergence has improved; when When this occurs, it indicates a decline in population convergence. Diversity status is assessed by the distribution uniformity index DV To assess and reflect the distribution of individuals in the target space: ; in, Indicates the Euclidean distance between adjacent individuals; Indicates the average distance; DV The smaller the value, the more evenly the population is distributed; ,when When, it indicates that population diversity has improved; when This indicates a decline in population diversity.

[0010] Preferably, based on generational distance GD and distribution uniformity index DV Based on the direction of change, the state space is defined as three discrete states: State 1: Both convergence and diversity are improved. and ; State 2: Any one improvement leads to another degradation: and or and ; State 3: Both convergence and diversity degenerate. and ; Rewards will be given based on the improvement of the discrete state. : .

[0011] Preferably, actions in S5 This represents 27 combinations consisting of 3 crossover strategies, 3 mutation strategies, and 3 neighborhood search strategies; Crossing strategies include two-point crossover, uniform crossover, and location-based crossover; Mutation strategies include bit-flip mutation, insertion mutation, and swap mutation; Neighborhood search strategies include critical process machine reallocation, critical process speed adjustment, and critical path process swapping; Critical process machine reassignment: Randomly select a process on the critical path, reassign it to another machine, and update the corresponding processing time; Speed ​​adjustment for critical processes: Randomly select a critical process, adjust the processing speed level, and recalculate the processing time and energy consumption indicators; Critical path process swapping: On the same machine, swap the processing order of two adjacent critical processes to optimize resource scheduling.

[0012] Preferably, in S9, by introducing new random individuals, the equilibrium is broken, helping the NSGA-II algorithm escape the local optimum region. Specifically, this includes the following operations: S91: Establish and maintain external archives: After each generation of NSGA-II evolution, add an archiving step to save extreme solutions; S92: Set up a stall detection mechanism: Mild stagnation indicates that the optimal solution is in continuous... No improvements were made within the generation, among which Indicates the maximum number of iterations; when a slight stagnation is detected, an external archive injection is triggered, randomly selecting an individual from the external archive to replace the worst-fitting individual in 10% of the total population. Severe stagnation indicates that the optimal solution is in continuous... No improvement was achieved within the generation; new individuals equivalent to 10% of the total population were randomly generated to replace the individuals with the worst fitness in the current population, and the process returned to step S4 to continue the iteration.

[0013] Therefore, the remanufacturing workshop scheduling method based on the fusion of reinforcement learning and evolutionary algorithms described above is adopted in this invention, and the technical effects are as follows: (1) This invention constructs a three-stage integrated scheduling model of disassembly-reprocessing-assembly, and introduces a multi-speed gear intelligent switching strategy for processing equipment, enabling the equipment to dynamically adjust its operating status according to the actual working conditions, and achieve the optimal balance between completion time and energy consumption. Taking into full account the actual situation of distributed heterogeneous factories, a multi-level decision-making mechanism of factory allocation, machine selection, speed selection and process sequencing is used to achieve efficient allocation of heterogeneous resources.

[0014] (2) The present invention adopts an intelligent optimization mechanism that integrates reinforcement learning and evolutionary algorithm. Through Q-learning, it dynamically selects the best search strategy to help the evolutionary algorithm NSGA-II intelligently search and select the optimal combination operator and neighborhood search strategy. With the help of adaptive parameter adjustment and dual escape mechanism, it effectively overcomes the defects of premature convergence and low search efficiency of traditional algorithms, achieves high-speed processing to shorten the cycle, low-speed operation to reduce energy consumption, and realizes intelligent trade-off.

[0015] (3) In this invention, the evolutionary algorithm NSGA-II successfully explored different regions of the trade-off space, greatly improving the quality and diversity of Pareto solutions, and its overall performance is better than that of traditional scheduling methods. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the remanufacturing system process of the present invention; Figure 2 This is a partial initialization diagram of the distributed heterogeneous flexible work workshop in an embodiment of the present invention; Figure 3 This is a comparison image of the Pareto front solution set in an embodiment of the present invention; Figure 4 This is a flowchart of the remanufacturing workshop scheduling method based on the fusion of reinforcement learning and evolutionary algorithm in this invention; Figure 5 This is a Gantt chart of the remanufacturing system in an embodiment of the present invention; Figure 6 This is a schematic diagram of the NSGA-II framework for the reinforcement learning fusion evolution algorithm in an embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0018] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0019] Example like Figure 4 As shown, this invention provides a remanufacturing shop scheduling method based on the fusion of reinforcement learning and evolutionary algorithms, including the following steps: S1: Construct a three-stage remanufacturing system model covering disassembly, distributed heterogeneous reprocessing, and assembly; like Figure 1 As shown, this paper studies the scheduling problem of a remanufacturing system comprising three stages: disassembly, reprocessing, and assembly. The system processes five scrap products (P1-P5), each product consisting of two core components (P1={j1,j2}, P2={j3,j4},...,P5={j9,j10}). These products undergo three consecutive stages: Dismantling stage: The waste products are dismantled into corresponding defective workpieces j. The dismantling workshop has two parallel dismantling machines D with the same processing time that can be selected.

[0020] Reprocessing stage: This stage takes place in a distributed system comprising three heterogeneous factories, with a total of six machines (Factory 1: M1, M2; Factory 2: M3, M4; Factory 3: M5, M6). The production capacity of the equipment in each factory varies, and the operation of each workpiece... Processing is carried out according to the selected machine.

[0021] Assembly stage: The remanufactured parts after the reprocessing process are assembled. Parts from the same product must be matched together during reassembly. The assembly workshop has two parallel assembly machines (A) to choose from. The product-part relationship is shown in Table 1. Table 1

[0022] S2: Define decision variables including factory selection, machine selection, process sequencing, and speed gear allocation; establish the maximum completion time, total energy consumption optimization objectives, and constraints; in S2, the maximum completion time... Represented as: ; In the formula, Indicates the maximum assembly completion time for the product; Total energy consumption function Represented as: ; In the formula, E1 Indicates the total energy consumption of dismantling; E2 Indicates the total energy consumption for reprocessing; E3 Indicates the total energy consumption of assembly; E4 Indicates the system's fixed energy consumption; ; The disassembly phase is a direct, no-wait process, so the total energy consumption of disassembly is equal to the product's energy consumption. P Disassembling the machine D On the processing time Multiply by unit power Decision variables Indicates product P Is the machine being disassembled? D Upgrade processing.

[0023] ; in, For processing energy consumption, workpiece j Operation i At the factory f machine k On the processing time Multiply by the corresponding unit processing power ; Idle energy consumption when factory machines are not in operation This refers to the unit idle power of the machine. Decision variable representation operation i Is it in the machine? k Upgrade processing.

[0024] Calculated processing time: ; in, For a fixed processing time of operation i for machine k, each machine can operate at multiple speed levels. Intelligent switching between speed levels (low level) ,intermediate ,advanced The higher the value, the shorter the actual processing time. The shorter the length, the greater the energy consumption will be. Choose a decision variable for speed, representing the operation. i Is it based on speed? In the machine k Execute above.

[0025] Total assembly energy consumption consists of assembly energy consumption and idle energy consumption of the assembly machine: ; Indicates product P Assembly machine A The assembly time and corresponding unit assembly power. , The unit idle power of the assembly machine. Decision variables represent products P Is it in the assembly machine? A Upper assembly.

[0026] ; The system's fixed total energy consumption is the fixed energy consumption of the entire remanufacturing process, which is the base energy consumption rate multiplied by the maximum completion time.

[0027] The constraints are expressed as follows: Each operation on a workpiece can only be performed on one machine: ; In the formula, This represents the decision variable, specifically the defective workpiece. j Operation i Is it in the machine? k Processing is performed; M represents a set of machines; Each operation can only select one machine of a certain speed class from the machine set: ; In the formula, Choose a decision variable for speed, representing the operation. i Is it based on speed? In the machine k Execute above.

[0028] Assembly workstations must perform assembly processes in sequence: ; Reprocessing must begin only after disassembly is complete: ; Assembly must begin after all parts have been reprocessed. ; S3: Four-dimensional encoding is used to represent the solution of the remanufacturing system model in S1, and a multi-strategy initialization method is used to generate the initial population; like Figure 2 As shown, to address the distributed heterogeneous characteristics of the reprocessing stage, this solution achieves fine optimization through a multi-layer intelligent decision-making mechanism: firstly, it dynamically allocates processes to the optimal factory based on process characteristics; secondly, it selects the most suitable processing machine within the factory; and finally, it matches the optimal speed level for each machine. In S3, the solution of the reprocessing system model is represented by a four-dimensional encoded process sorting vector, factory allocation vector, machine allocation vector, and speed selection vector. The proportion of the initialization strategy in the initialization stage is adjusted to generate an initialization population. The specific allocation strategy is expressed as follows: The shortest processing time priority rule accounts for 40%, prioritizing tasks with short processing times. For each processing step, the available machine with the shortest processing time is selected, and high-speed processing (highest speed level 1.2) is used to further compress processing time and optimize the completion time target. Intelligent allocation strategies based on factory machine resource configuration account for 40%, prioritizing factories with the most idle machines; and rationally allocating processes to relatively idle machines to improve overall equipment utilization.

[0029] Random initialization accounts for 20%, generating solutions entirely randomly. This process provides an efficient initialization solution for optimizing multi-objective remanufacturing scheduling systems.

[0030] S4: An adaptive method is used to dynamically adjust the crossover probability and mutation probability according to the algorithm's search progress; the calculation methods for the crossover probability pc and mutation probability pm in S4 are as follows: ; ; ; In the formula, and The factor representing the regulation of crossover and mutation has a value range of (0,1]. N The number of solutions in the current Pareto front; Indicates the first p The objective function at the th ... q The value on each solution; Indicates the first p In the objective function, with the first q The target value of the individual whose solution is closest to the target value; This describes the total dispersion of all solutions on the current Pareto front; This represents the relative proportion between the current solution and the optimal solution; denoted by the natural constant; the formula pm indicates that if the objective function value of the current solution is far from the optimal value, the mutation probability is increased, enhancing the ability to escape local optima.

[0031] S5: As Figure 6 As shown, the Q-learning-based intelligent decision-making mechanism dynamically selects the optimal combination of crossover operator, mutation operator, and neighborhood search strategy for the NSGA-II algorithm based on the real-time search state; the core decision-making mechanism is based on the following formula: ; in For learning rate, The discount rate is set to a value range of (0,1]. Indicates the state Select action Current value estimate, Provides immediate performance feedback for the current action; Indicates the state Next action The highest expected Q value that can be obtained from the Q-value table; In S5, by continuously iterating and updating the Q-value table, the Q-learning algorithm gradually learns the value mapping of the optimal operator combination under different search states and establishes the optimal decision strategy. This intelligent decision-making mechanism based on Q-learning enables the NSGA-II algorithm to have the ability to adaptively adjust the search direction. It can dynamically select the optimal combination of crossover, mutation and local search strategies according to the real-time search state, which significantly improves the solution efficiency and solution set quality of multi-objective optimization problems.

[0032] state Assess the convergence and diversity of the current Pareto frontier; Convergence state ΔGD Using generational distance GD As a convergence evaluation metric, GD The Pareto solution set that measures the current population P With reference to Pareto Frontier The average minimum distance between them is calculated using the following formula: ; in, Indicates population P The Middle q Individual to reference set Euclidean distance to the nearest individual; GD The smaller the value, the better the convergence of the population; during the iteration process, the change in the convergence state is defined as: ;when When, it indicates that the population convergence has improved; when When this occurs, it indicates a decline in population convergence. Diversity status is assessed by the distribution uniformity index DV To assess and reflect the distribution of individuals in the target space: ; in, Indicates the Euclidean distance between adjacent individuals; Indicates the average distance; DV The smaller the value, the more evenly the population is distributed; ,when When, it indicates that population diversity has improved; when This indicates a decline in population diversity.

[0033] Based on generational distance GD and distribution uniformity index DV Based on the direction of change, the state space is defined as three discrete states: State 1: Both convergence and diversity are improved. and ; State 2: Any one improvement leads to another degradation: and or and ; State 3: Both convergence and diversity degenerate. and ; Rewards will be given based on the improvement of the discrete state. : .

[0034] Actions in S5 This represents 27 combinations consisting of 3 crossover strategies, 3 mutation strategies, and 3 neighborhood search strategies; Crossing strategies include two-point crossover, uniform crossover, and location-based crossover; Mutation strategies include bit-flip mutation, insertion mutation, and swap mutation; Neighborhood search strategies include critical process machine reallocation, critical process speed adjustment, and critical path process swapping; Critical process machine reassignment: Randomly select a process on the critical path, reassign it to another machine, and update the corresponding processing time; Speed ​​adjustment for critical processes: Randomly select a critical process, adjust the processing speed level, and recalculate the processing time and energy consumption indicators; Critical path process swapping: On the same machine, swap the processing order of two adjacent critical processes to optimize resource scheduling.

[0035] S6: Perform crossover, mutation, and local search strategies on the quasi-groups in S3; S7: Perform fast non-dominated sorting on the population obtained in S6, calculate the crowding degree, and generate a new generation of population based on the elite preservation strategy. S8: Determine if the current generation has reached the maximum number of iterations. If the result is yes, end the iteration and output the optimal scheduling scheme; if the result is no, proceed to the next step S9. S9: Execute the escape mechanism, introduce new random individuals to help the NSGA-II algorithm escape the local optimum region, return to S4 to continue iterating until the maximum number of iterations is reached and the optimal scheduling scheme is output.

[0036] In S9, by introducing new random individuals, the equilibrium is broken, helping the NSGA-II algorithm escape local optima. This includes the following operations: S91: Establish and maintain external archives: After each generation of NSGA-II evolution, add an archiving step to save extreme solutions; S92: Set up a stall detection mechanism: Mild stagnation indicates that the optimal solution is in continuous... No improvements were made within the generation, among which Indicates the maximum number of iterations; when a slight stagnation is detected, an external archive injection is triggered, randomly selecting an individual from the external archive to replace the worst-fitting individual in 10% of the total population. Severe stagnation indicates that the optimal solution is in continuous... No improvement was achieved within the generation; new individuals equivalent to 10% of the total population were randomly generated to replace the individuals with the worst fitness in the current population, and the process returned to step S4 to continue the iteration.

[0037] Comparative experiment As shown in Table 2, key parameters were set uniformly, and different comparison algorithms all used the same number of iterations of 3000 and population size of 100. Table 2

[0038] The comparison results of the objective functions are shown in Table 3: Table 3

[0039] The comparison of the evaluation index values ​​is shown in Table 4;

[0040] Tables 3 and 4 show the overall performance comparison between this scheme and five mainstream multi-objective algorithms under the same hardware and software environment. All results are the average of 10 independent runs.

[0041] As shown in Table 3, the optimization results of the core objectives demonstrate that this scheme achieved optimal values ​​in both maximum processing time and total energy consumption. Specifically, the maximum processing time of 1659.8 seconds was reduced by 2.4% to 5.3% compared to the comparative algorithms, directly reflecting the significant advantages of the three-stage collaborative scheduling and intelligent strategy in compressing the production cycle. Regarding energy consumption, the total energy consumption of 144.3 kWh was reduced by 0.3% to 5.0% compared to other algorithms, verifying the effectiveness of the multi-speed gear intelligent switching strategy in achieving refined energy efficiency management and successfully solving the trade-off between production efficiency and energy consumption.

[0042] IGD (Inverse Generation Distance) measures the distance from the reference point on the real Pareto front to the solution set obtained by the algorithm. It indicates the degree of approximation and coverage of the algorithm's solution set to the real Pareto front. As shown in Table 4, the IGD value of this scheme is the smallest, indicating that the obtained solution set is closer to the real Pareto front and has a more uniform distribution. It is superior to the comparison algorithms in terms of convergence and diversity.

[0043] HV (Hypervolume): Measures the volume of the target space bounded by the non-dominated solution set obtained by the algorithm and the reference point, reflecting the convergence, diversity, and coverage of the solution set. Compared with other algorithms, this scheme has the largest HV value, indicating that the solution set dominates a larger area in the target space and has better overall performance.

[0044] NPS (Number of Non-Dominated Solutions): If the number of solutions is too small, it indicates that the solutions may be crowded into a very small region, failing to represent the entire trade-off space and provide decision-makers with a sufficient number of options. This approach yields the largest number of solutions, indicating that the number of solutions obtained is efficient and non-redundant. It successfully explores different regions of the trade-off space and finds a large number of high-quality non-dominated solutions.

[0045] The Pareto front comparison images of this scheme with other algorithms are as follows: Figure 3 As shown, it is evident that the Pareto solutions obtained by this scheme are numerous and relatively dispersed, and the two objectives of maximum processing time and energy consumption remain at a relatively low level, verifying the effectiveness and superiority of the algorithm in solving remanufacturing problems.

[0046] The Gantt chart generated by this solution is as follows: Figure 5 As shown, this system achieves full-process visual monitoring of a distributed heterogeneous remanufacturing system. It visualizes the collaborative scheduling process of heterogeneous resources across multiple factories, displaying not only traditional time-based scheduling but also intuitively presenting intelligent speed control strategies through multi-speed level identifiers (such as 'v1.2'). The task blocks corresponding to different speeds visually reflect the time-energy consumption trade-off decisions, demonstrating the crucial role of intelligent speed control in energy efficiency optimization. This achieves shorter processing cycles with high speeds and reduced energy consumption with low-speed operation, realizing an intelligent trade-off.

[0047] Therefore, this invention adopts the above-mentioned remanufacturing workshop scheduling method based on the fusion of reinforcement learning and evolutionary algorithm. By using the hybrid mechanism of reinforcement learning-assisted evolutionary algorithm and combining it with multi-speed gear strategy, it effectively solves the coordination problem of resource allocation and energy consumption control in a distributed environment. Through the cascade optimization of factory-machine-speed, it significantly improves energy utilization while ensuring production efficiency, highlighting the intelligent advantages of this solution in dealing with heterogeneous resources and multi-objective balance. Its comprehensive performance is significantly better than existing solutions.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A remanufacturing shop scheduling method based on the fusion of reinforcement learning and evolutionary algorithms, characterized in that, Includes the following steps: S1: Construct a three-stage remanufacturing system model covering disassembly, distributed heterogeneous reprocessing, and assembly; S2: Define decision variables including factory selection, machine selection, process sequencing and speed level allocation, and establish the optimization objectives and constraints for maximum completion time and total energy consumption; S3: Four-dimensional encoding is used to represent the solution of the remanufacturing system model in S1, and a multi-strategy initialization method is used to generate the initial population; S4: Use an adaptive method to dynamically adjust the crossover probability and mutation probability according to the search progress of the algorithm; S5: A Q-learning-based intelligent decision-making mechanism that dynamically selects the optimal combination of crossover, mutation, and neighborhood search strategies for the NSGA-II algorithm based on the real-time search status; the core decision-making mechanism is based on the following formula: ； in For learning rate, The discount rate is set to a value range of (0,1]. Indicates the state Select action Current value estimate, Provides immediate performance feedback for the current action; Indicates the state Next action The highest expected Q value that can be obtained from the Q-value table; S6: Perform crossover, mutation, and local search strategies on the quasi-groups in S3; S7: Perform fast non-dominated sorting on the population obtained in S6, calculate the crowding degree, and generate a new generation of population based on the elite preservation strategy. S8: Determine if the current generation has reached the maximum number of iterations. If the result is yes, end the iteration and output the optimal scheduling scheme; if the result is no, proceed to the next step S9. S9: Execute the escape mechanism, introduce new random individuals to help the NSGA-II algorithm escape the local optimum region, return to S4 to continue iterating until the maximum number of iterations is reached and the optimal scheduling scheme is output.

2. The remanufacturing shop scheduling method based on the fusion of reinforcement learning and evolutionary algorithms according to claim 1, characterized in that, Maximum completion time in S2 Represented as: ； In the formula, Indicates the maximum assembly completion time for the product; Total energy consumption function Represented as: ； In the formula, E1 Indicates the total energy consumption of dismantling; E2 Indicates the total energy consumption for reprocessing; E3 Indicates the total energy consumption of assembly; E4 Indicates the system's fixed energy consumption; The constraints are expressed as follows: Each operation on a workpiece can only be performed on one machine: ； In the formula, This represents the decision variable, specifically the defective workpiece. j Operation i Is it in the machine? k Processing is performed; M represents a set of machines; Each operation can only select one machine of a certain speed class from the machine set: ； In the formula, Choose a decision variable for speed, representing the operation. i Is it based on speed? In the machine k Execute above.

3. The remanufacturing shop scheduling method based on the fusion of reinforcement learning and evolutionary algorithms according to claim 1, characterized in that, In S3, the solution of the earlier system model is represented by a four-dimensional encoded process sorting vector, factory allocation vector, machine allocation vector, and speed selection vector. The proportion of the initialization strategy in the initialization phase is allocated to generate the initialization population. The specific allocation strategy is expressed as follows: The shortest processing time priority rule accounts for 40%, prioritizing tasks with shorter processing times and optimizing completion time targets; Intelligent allocation strategies based on factory machine resource allocation account for 40%, prioritizing factories with the most idle machines. Random initialization accounts for 20%, and solutions are generated randomly.

4. The remanufacturing shop scheduling method based on the fusion of reinforcement learning and evolutionary algorithms according to claim 1, characterized in that, The crossover probability pc and mutation probability pm in S4 are calculated as follows: ； ； ； In the formula, and The factor representing the regulation of crossover and mutation has a value range of (0,1]. N The number of solutions in the current Pareto front; Indicates the first p The objective function at the th ... q The value on each solution; Indicates the first p In the objective function, with the first q The target value of the individual whose solution is closest to the target value; This describes the total dispersion of all solutions on the current Pareto front; This represents the relative proportion between the current solution and the optimal solution; Represents the natural constant.

5. The remanufacturing shop scheduling method based on the fusion of reinforcement learning and evolutionary algorithms according to claim 1, characterized in that, In S5, by continuously updating the Q-value table, the Q-learning algorithm gradually learns the value mapping of the optimal operator combination under different search states and establishes the optimal decision-making strategy. state Assess the convergence and diversity of the current Pareto frontier; Convergence state ΔGD Using generational distance GD As a convergence evaluation metric, GD The Pareto solution set that measures the current population P With reference to Pareto Frontier The average minimum distance between them is calculated using the following formula: ; in, Indicates population P The Middle q Individual to reference set Euclidean distance to the nearest individual; GD The smaller the value, the better the convergence of the population; during the iteration process, the change in the convergence state is defined as: ;when When, it indicates that the population convergence has improved; when When this occurs, it indicates a decline in population convergence. Diversity status is assessed by the distribution uniformity index DV To assess and reflect the distribution of individuals in the target space: ; in, Indicates the Euclidean distance between adjacent individuals; Indicates the average distance; DV The smaller the value, the more evenly the population is distributed; ,when When, it indicates that population diversity has improved; when This indicates a decline in population diversity.

6. The remanufacturing shop scheduling method based on the fusion of reinforcement learning and evolutionary algorithms according to claim 5, characterized in that, Based on generational distance GD and distribution uniformity index DV Based on the direction of change, the state space is defined as three discrete states: State 1: Both convergence and diversity are improved. and ; State 2: Any one improvement leads to another degradation: and or and ; State 3: Both convergence and diversity degenerate. and ; Rewards will be given based on the improvement of the discrete state. : 。 7. The remanufacturing shop scheduling method based on the fusion of reinforcement learning and evolutionary algorithms according to claim 1, characterized in that, Actions in S5 This represents 27 combinations consisting of 3 crossover strategies, 3 mutation strategies, and 3 neighborhood search strategies; Crossing strategies include two-point crossover, uniform crossover, and location-based crossover; Mutation strategies include bit-flip mutation, insertion mutation, and swap mutation; Neighborhood search strategies include critical process machine reallocation, critical process speed adjustment, and critical path process swapping; Critical process machine reassignment: Randomly select a process on the critical path, reassign it to another machine, and update the corresponding processing time; Speed ​​adjustment for critical processes: Randomly select a critical process, adjust the processing speed level, and recalculate the processing time and energy consumption indicators; Critical path process swapping: On the same machine, swap the processing order of two adjacent critical processes to optimize resource scheduling.

8. The remanufacturing shop scheduling method based on the fusion of reinforcement learning and evolutionary algorithms according to claim 1, characterized in that, In S9, by introducing new random individuals, the equilibrium is broken, helping the NSGA-II algorithm escape local optima. This includes the following operations: S91: Establish and maintain external archives: After each generation of NSGA-II evolution, add an archiving step to save extreme solutions; S92: Set up a stall detection mechanism: Mild stagnation indicates that the optimal solution is in continuous... No improvements were made within the generation, among which Indicates the maximum number of iterations; When a slight stagnation is detected, an external archive injection is triggered, randomly selecting an individual from the external archive to replace the worst-fit individuals in 10% of the total population. Severe stagnation indicates that the optimal solution is in continuous... No improvement was achieved within the generation; new individuals equivalent to 10% of the total population were randomly generated to replace the individuals with the worst fitness in the current population, and the process returned to step S4 to continue the iteration.

Images