Intelligent job shop scheduling method

Abstract

The invention discloses an intelligent job shop scheduling method. The method comprises the following steps: S1, establishing a personalized improved NSGA-II algorithm module and a DQN-based adaptive decision algorithm module; s2, establishing a mathematical model taking the maximum completion time, the total load of equipment and the total energy consumption of a workshop as optimization targets; s3, in a personalized improved NSGA-II algorithm module, considering a workshop optimization target, adopting multiple strategies for initialization, and adding a user personalized demand to screen a target population; s4, in a DQN-based adaptive decision algorithm module, population features are extracted to construct an agent state space, and a deep Q network experience playback mechanism is utilized to train the relationship between agent capture scheduling information and a scheduling target to perform adaptive action decision; and S5, returning the optimal action predicted by the DQN model to the NSGA-II, adjusting the optimal parameter proportion according to the real-time state, and improving the multi-target solution quality.

Description

Technical Field

[0001] The present invention relates to the technical field of scheduling and reinforcement learning of flexible job shops, and in particular to an intelligent job shop scheduling method. Background Art

[0002] Compared with the Industry 4.0 paradigm, Industry 5.0 emphasizes respecting and protecting the earth's ecology while pursuing production efficiency and technological innovation, and fully considering human needs and values, indicating that the manufacturing industry is gradually shifting from a technology-driven model to a resilient, sustainable and people-oriented value-driven model [Ivanov D. The Industry 5.0 framework: viability-based integration of the resilience, sustainability, and human-centricity perspectives [J]. International Journal of Production Research, 2023, 61 (5): 1683-1695.]. The current research adjustment is how to ensure production efficiency while taking into account the welfare of workers and the needs of enterprises to establish a sustainable development environment.

[0003] "People-oriented" is one of the core principles of Industry 5.0, which puts the core needs of human beings and the well-being of workers at the center of the production process, emphasizes human-machine collaboration, improves production efficiency, promotes innovation and personalized production, and meets the market demand for diversified and customized products. For example, the article [Rinaldi M, Fera M, Bottani E, Grosse EH. Workforce scheduling incorporating worker skills and ergonomic constraints [J]. Computers & Industrial Engineering, 2022, 168: 108107.] takes the skill level of workers into consideration and constructs a mathematical model of worker performance to optimize labor allocation and improve performance. In this paper, we further extend the concept of people-oriented and apply it to workshop scheduling practice, allowing enterprises to customize and set production goals according to their own goals and preferences, and the scheduling method can flexibly adapt to changes in demand.

[0004] Under the framework of Industry 5.0, the principle of sustainability focuses on resource conservation. The energy consumption of the industrial sector accounts for 31.7% of the world's total. Achieving sustainability in the manufacturing industry is a major challenge facing environmental protection. The article [Chaurasiya SK, Singh G. Exploring Sustainable Manufacturing: A Comprehensive Review of Literature and Practices [C] / / International Conference on Production and Industrial Engineering. Singapore: Springer Nature Singapore, 2023: 11-19.] explores the potential of optimizing waste treatment and circular economy, and comprehensively analyzes the role of technology and digitalization in promoting sustainable manufacturing. In the job shop scheduling problem, the concept of sustainability is manifested in reducing waste emissions and reducing workshop energy consumption during the production process. Through the intelligent scheduling system, the production process and operation sequence are optimized, and the idle time of the machine is reduced, thereby reducing non-essential energy consumption, so as to achieve a win-win situation for environmental and economic benefits.

[0005] The methods for solving job-shop scheduling problems can be roughly divided into two categories: exact methods and approximate methods. Exact methods mainly use operations research methods to find the optimal solution, but they require the design of more complex constraints and are not suitable for solving large-scale practical problems. In order to focus on practical applications, an approximate method is proposed to solve large-scale scheduling problems. The article [Zhang H, Xu G, Pan R, et al. A novel heuristic method for the energy-efficient flexible job-shop scheduling problem with sequence-dependent set-up and transportation time [J]. Engineering Optimization, 2022, 54 (10): 1646-1667.] proposes a new heuristic method for large-scale problems, but since heuristic methods are usually based on simple decision rules, their exploration capabilities are limited. To solve the above problems, intelligent optimization algorithms are proposed, including genetic algorithms, differential evolution algorithms, artificial bee colony algorithms, etc. For example, the article [Ding H, Gu X. Improved particle swarm optimization algorithm based novel encoding and decoding schemes for flexible job shop scheduling problem [J]. Computers & Operations Research, 2020, 121: 104951.] introduces molecular forces and proposes a particle swarm optimization algorithm based on molecular motion theory, which can effectively balance global and local searches. Although the intelligent optimization algorithm can obtain high-quality solutions, it is sensitive to parameter settings and has poor versatility. To solve the problems of the intelligent optimization algorithm, reinforcement learning technology is applied to the job shop scheduling problem. For example, the article [Luo S. Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning [J]. Applied Soft Computing, 2020, 91: 106208.] proposes a two-layer deep Q network online rescheduling framework for real-time optimization scheduling decisions. Due to the high correlation between the characteristics of this problem, it may interfere with the neural network and increase the computational burden.To further explore the application of model fusion in job shop scheduling problems, the article [Lei K, Guo P, Zhao W, Wang Y, Qian L, Meng X, Tang LA multi-action deep reinforcement learning framework for flexible Job-shop scheduling problem [J]. Expert Systems with Applications, 2022, 205: 117796.] proposes a multi-pointer graph network architecture and a multi-neighbor strategy optimization training algorithm to improve generalization performance. The model fusion method achieves Makespan minimization and improves solution efficiency, making significant progress in the high efficiency requirements of Industry 4.0. However, the consideration of environmental sustainability is still insufficient. In response to Industry 5.0's focus on sustainability, the article [Rui Z, Zhang X, Liu M, Ling L, Wang X, Liu C, Sun M. Graph reinforcement learning for flexible job shop scheduling under industrial demand response: A production and energy nexus perspective [J]. Computers & Industrial Engineering, 2024, 193: 110325.] uses graph neural networks and deep reinforcement learning to optimize production scheduling, optimizing production efficiency and energy targets. However, the above methods all ignore the company's preference needs for optimization targets and do not reflect the focus on personalized needs in Industry 5.0.

[0006] In summary, traditional workshop scheduling methods focus more on improving production efficiency, but ignore human and environmental factors. Industry 5.0 emphasizes the integration of personalization and sustainability requirements while improving efficiency. Summary of the invention

[0007] The purpose of the present invention is to provide an intelligent job shop scheduling method. The method includes an improved NSGA-Ⅱ algorithm module based on personalization and an adaptive decision algorithm module based on DQN. Among them, the improved NSGA-Ⅱ algorithm module based on personalization considers efficiency and energy consumption targets, adopts multiple strategies for population initialization, introduces an evolutionary mechanism based on the individual needs of the enterprise to screen the population, and continuously screens out approximate optimal solutions through operations such as crossover and mutation of genes. Based on the DQN adaptive decision algorithm module, the state space is constructed by extracting population and workshop characteristics, and the intelligent agent is trained using empirical information. The relationship between the population state and the scheduling target is captured to make action decisions, the optimal proportion parameters are dynamically adjusted, and the space is explored for better solutions. While meeting the individual preference needs of the enterprise, the efficiency target and the sustainability target are optimized to a greater extent.

[0008] The technical solution adopted by the present invention is:

[0009] An intelligent job shop scheduling method comprises the following steps:

[0010] S1: Build an improved NSGA-Ⅱ algorithm module based on personalization and an adaptive decision algorithm module based on DQN;

[0011] S2: Establish to minimize the maximum completion time C max 、Total equipment load L all And the total energy consumption of the workshop E all To optimize the book order model of the target, design relevant constraints according to the processing constraint characteristics of the flexible job shop;

[0012] S3: Based on the personalized improved NSGA-Ⅱ algorithm module, multi-strategy initialization and dual-strategy selection mechanism are used to optimize the solution quality. Through data normalization technology, the search bias caused by the difference in target magnitude is eliminated, and personalized settings for enterprise needs are added to screen the target population;

[0013] S4: In the DQN-based module for adaptively adjusting key parameters, the static characteristics of the workshop and the dynamic characteristics of the population are extracted to construct the state space of the agent. The deep Q network experience replay mechanism is used to train the agent to capture the relationship between scheduling information and scheduling goals for adaptive action decision-making;

[0014] S5: Return the best action predicted by the DQN model to NSGA-II, adjust the optimal parameter ratio according to the real-time status, and improve the quality of multi-objective solution.

[0015] Furthermore, in step S3, a multi-strategy initialization and dual-strategy selection mechanism is used to optimize the solution quality, and the search bias caused by the target magnitude difference is eliminated through data normalization technology, and then personalized settings for enterprise needs are added to screen and obtain the target population, including the following steps:

[0016] 1-1) Double-layer real number coding is adopted, including "process sorting" and "machine selection". The chromosome length should be Where n represents the total number of workpieces, n i Indicates the number of processes contained in the workpiece;

[0017] 1-2) Decode the encoding scheme using plug-in greedy decoding. The specific decoding principles are as follows:

[0018] If process O ij is the first process of the workpiece, and O ij For machine M k Select the first process to be processed and process it directly at time zero;

[0019] If process O ij It's Machine M k The first process to be processed is in the previous process O i(j-1) Process the process at the time of completion of processing;

[0020] If not, in the machine M k Find the idle segments between the processing steps in turn [ts k ,te k ], calculation process O ij Earliest start time of processing ijk , determine whether the idle time can meet the processing time t required by the process ijk , and satisfy the constraints, process O ij Insert this time period; if the previous process O i(j-1) The processing completion time is less than ts k , process O ij The start processing time is ts k , otherwise the starting processing time is process O i(j-1) The processing completion time;

[0021] If the idle time does not meet the process O ij Processing conditions, then continue to look for the next idle segment [ts k+1 ,te k+1 ];

[0022] 1-3) In order to obtain a better initial solution, multiple strategies are used to initialize the population, including random initialization RS, global minimum machine energy consumption GES, global minimum processing time GS, and local minimum processing time LS. The population is initialized according to the proportion of 20%, 30%, 30%, and 20%;

[0023] 1-4) Use elite selection and binary league to select individuals to be crossed, and design random number selection:

[0024] In the early stage of iteration, if it is less than this random number, two individuals are selected from the parent population and the champion pool of the binary league; if it is greater than this random number, two individuals are selected from the parent population and the champion pool of the elite pool;

[0025] In the later stage of iteration, if it is less than this random number, two individuals are selected from the parent population and the elite pool; if it is greater than this random number, two individuals are selected from the elite pool and the champion pool of the binary league;

[0026] 1-5) Design different crossover and mutation operators:

[0027] Process crossover operator: IPOX is used; the job set is randomly divided into two mutually exclusive subsets, and the job numbers and corresponding machine codes of different parent chromosomes are cross-assigned to the two child chromosomes;

[0028] Machine crossover operator: PMX and SBX are used; the PMX algorithm randomly determines a subset to initialize the exchange process, then screens the job numbers belonging to the subset, and only exchanges the machine codes that meet the conditions, thereby achieving genetic recombination of some characteristics between parent chromosomes; the SBX algorithm initially defines the job subset and generates a control parameter β to regulate the crossover process, and mixes the parent chromosome fragments of the jobs in the subset according to the formula to form a new offspring, where p1 ms 、p2 ms Respectively represent the two parent machine number parts;

[0029] p1 ms =0.5×((1+β)×p1 ms +(1-β)×p2 ms );

[0030] p2 ms =0.5×((1+β)×p2 ms +(1-β)×p1 ms );

[0031] Process mutation operator: exchange mutation and polygon mutation are used; exchange mutation is achieved by exchanging the positions of the job number and its corresponding machine code in the chromosome; polygon mutation uses a random number generation strategy to determine the mutation probability of each gene in the chromosome. When the random number is lower than the predetermined mutation probability threshold, another position on the chromosome is randomly selected for gene exchange;

[0032] Machine mutation operator: single-point mutation is used; a gene site in the chromosome is randomly selected, and the process is determined according to the corresponding gene code; a machine is randomly selected from the list of optional machines in the process to achieve mutation update of the corresponding machine code;

[0033] 1-6) Convert the target importance into optimization priority as the initial input before scheduling. The subsequent steps are as follows:

[0034] Step 1: For each objective function f x Data normalization is performed to eliminate the impact of different dimensions and value ranges, ensuring that the algorithm does not favor targets with larger values ​​during the search process. The processing formula is as follows;

[0035]

[0036] where g x is the function value after normalization, min(f) and max(f) represent the minimum and maximum values ​​of the objective function respectively;

[0037] Step 2: Pareto stratify the population according to the three objective function values. In each stratum, individuals will be sorted according to the specific needs of the enterprise, and those solutions with high priority and low objective function values ​​will be selected first to ensure that they can be selected into the new population first. If the objective function values ​​are the same, the crowding distance will be used to screen the population to avoid the aggregation of solutions.

[0038] Step 3: To prevent only the low non-dominated level from being selected, select the non-dominated individuals in proportion to enter the offspring population, while retaining the elite solution and maintaining the diversity of the population. The selection ratio is calculated as follows:

[0039] num v =min(sp×rank v ,pop size -pop now );

[0040] Among them, num v Indicates the number of individuals selected in the current non-dominated layer, rank v Indicates the number of individuals in the current layer, sp is the selection percentage, pop size is the population size, pop now The current number of children.

[0041] Furthermore, in step S4, in the DQN adaptive decision algorithm module, the static characteristics of the workshop and the dynamic characteristics of the population are extracted to construct the state space of the agent, and the deep Q network experience playback mechanism is used to train the agent to capture the relationship between the scheduling information and the scheduling target to make adaptive action decisions, including the following steps:

[0042] 2-1) Extract static workshop features and dynamic population features from the iteratively generated population information, where the static features include the total number of workpieces N_num, the total number of equipment M_num, and the total number of processes of all workpieces Ni_total;

[0043] The following six dynamic features are extracted:

[0044] Uniformity: describes the uniform distribution of the current Pareto frontier solution of the population. The specific calculation is as follows:

[0045]

[0046] Among them, S describes uniformity, |A| represents the number of Pareto frontier solutions, d represents the average level of the current Pareto frontier solution distance, and d v is the solution distance between the vth individual and other individuals, L represents the number of objective functions, f l (x v ) and f l (x u ) represent the lth objective function values ​​of the vth individual and the uth individual respectively;

[0047] Solution quality GD: describes the degree of proximity of all individuals in the population to the current Pareto optimal frontier. The specific definition formula is as follows:

[0048]

[0049] Among them, pop represents the population size, ||·||2 represents the L2 norm of the Euclidean distance, and p v .f(x) and pareto.f(x) represent the fitness vectors of the individual and the optimal frontier individual respectively;

[0050] Fitness: the average of the three objectives of the lowest-ranked individual in the non-dominated layer of the current population, and the fitness value of the best individual;

[0051] The action space is constructed by dividing the crossover and mutation parameters with a certain step length. The crossover parameters are divided and generated between 0.5 and 0.9, and the mutation parameters are divided and generated between 0.1 and 0.4. The divided parameters are combined and arranged to generate the action space. The agent selects actions after network training to determine the key parameters P in the current iteration. c , P m ;

[0052] 2-2) The reward function is designed by weighting the impact of different input values ​​in the state space, taking into account the changes in multiple key indicators, giving significant rewards for improved behavior and imposing penalties for declining indicators. The reward weighted calculation is as follows:

[0053] R=ω1R1+ω2R2+…+ω n R n ;

[0054] Among them, ω n Corresponding to different state weights, Rn Corresponding to the difference of different state changes;

[0055] When the key indicators decrease at the same time, it is considered that the agent may have approached or reached the optimal solution. At this time, an additional significant reward is given and the termination state d is set. s =1;

[0056] 2-3) Adopt ε-greedy strategy for action selection, batch action selection mechanism. In the exploration phase, the agent randomly selects a batch of actions from the action set. In the utilization phase, the agent selects the top 6 actions with the highest Q value according to the Q value output by the model. ε represents the greedy rate, and the initial value is set to 0.9. The strategy selection is as follows:

[0057]

[0058] The exponential decay is used to adjust the exploration rate and control the decay speed of ε. The specific calculation is as follows:

[0059] ε=min_ε+(max_ε-min_ε)*e -LAMBDA×step ;

[0060] Among them, step is the training time step, LAMBDA is the decay rate, min_ε and max_ε represent the minimum and maximum greedy rates respectively;

[0061] 2-4) The DQN algorithm is used for model training, which includes two networks with the same structure, the Q value network and the Target Q network, which are uniformly constructed as a three-layer fully connected neural network. The ReLU function is used as the activation function to introduce nonlinear characteristics to the network. The specific process is as follows:

[0062] Step 1: Initialize the Q value network and Target Q network parameters, and define the experience buffer to store the state transition tuple Q(s t ,a t ,r t ,s t+1 ,d t ), including the state s t 、Action a t , Reward t , next state s t+1 And the completion mark d t , using fixed parameters to guide population crossover and mutation, this process randomly selects actions in batches, collects experience information and stores it in the experience memory bank, providing a sample basis for subsequent network training;

[0063] Step 2: Randomly extract a batch of data from the experience memory library as training samples, use the behavior value network to predict the Q(s, a) of all actions in the current state, and use the Target Q network to predict the different Q values ​​of the next state, and select the maximum Q value corresponding to each state;

[0064] Step 3: Calculate the target Q value by combining the immediate reward and the future reward. The calculation method is as follows:

[0065] Q(s,a)←Q(s,a)+α(R t+1 +γmax a' Q(s',a')(1-d s ));

[0066] Among them, γ is the discount factor that controls the importance of immediate rewards and experience rewards, α is the learning rate that controls the trade-off between new and old estimates, and d s Indicates whether the current state is a terminal state. For a terminal state, the target Q value is set to 0 to reflect that no further exploration should be done in the terminal state.

[0067] Step 4: To stabilize the learning process and reduce instability during training, the network parameters θ' are soft-updated. In each training, the parameters of the Q value network are moved to the parameters of the Target Q network according to the time step to smooth the learning process:

[0068] t new =t old ×(1-TAU)+e old ×TAU;

[0069] Among them, t old ,t new is the Target Q network parameter before and after the update, e old is the Q value network parameter, and TAU is a hyperparameter that controls the update speed.

[0070] The beneficial effects of adopting the above technical solution are:

[0071] The purpose of the present invention is to provide an intelligent job shop scheduling method, which includes an improved NSGA-Ⅱ algorithm module based on personalization and an adaptive decision algorithm module based on DQN. Among them, the improved NSGA-Ⅱ algorithm module based on personalization considers efficiency and energy consumption targets, adopts multiple strategies for population initialization, introduces an evolutionary mechanism based on the individual needs of the enterprise to screen the population, and continuously screens out approximate optimal solutions through operations such as crossover and mutation of genes. Based on the DQN adaptive decision algorithm module, the state space is constructed by extracting population and workshop characteristics, and the intelligent agent is trained using empirical information. The relationship between the population state and the scheduling target is captured to make action decisions, the optimal proportion parameters are dynamically adjusted, and the space is explored for better solutions. While meeting the individual preference needs of the enterprise, the efficiency target and the sustainability target are optimized to a greater extent.

[0072] The intelligent job shop scheduling method proposed in this paper is experimentally verified by 24 examples in the public benchmarks Brandimarte (MK series), Hurink (ABZ, LA, ORB), and GE. Two typical enterprise demand scenarios are set to verify the scheduling ability of the method under the individual needs of the enterprise. Compared with the mainstream algorithm, the maximum completion time is reduced by at least 2.5% and at most 12%, the total equipment load is reduced by at least 0.7% and at most 5.9%, and the total energy consumption of the workshop is reduced by at least 2.4% and at most 13%. It is verified that the proposed method takes into account the personalized and sustainable manufacturing needs while ensuring production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 This is a framework diagram of the intelligent job shop scheduling method;

[0074] Figure 2 This is the module framework diagram of the improved NSGA-Ⅱ algorithm based on personalization;

[0075] Figure 3 It is a schematic diagram of the decoding principle;

[0076] Figure 4 Schematic diagram of population selection for dual-strategy mixture;

[0077] Figure 5 This is a module framework diagram based on the DQN adaptive decision-making algorithm;

[0078] Figure 6 It is a network structure diagram;

[0079] Figure 7 Iteration diagram of objective function for case GE80-5;

[0080] Figure 8 Iteration diagram of objective function for case MK08;

[0081] Fig. 9 It is the experimental comparison diagram under different examples;

[0082] Fig.10 HV mean curves of different examples. DETAILED DESCRIPTION

[0083] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0084] The present invention takes flexible job shop scheduling as the background, reinforcement learning as the carrier, and uses the personalized improved NSGA-Ⅱ algorithm module and the DQN-based adaptive decision algorithm module as the main algorithm framework. Figure 1 As shown, the following steps are included:

[0085] S1: Build an improved NSGA-Ⅱ algorithm module based on personalization and an adaptive decision algorithm module based on DQN;

[0086] S2: Establish to minimize the maximum completion time C max 、Total equipment load L all And the total energy consumption of the workshop E all To optimize the book order model of the target, design relevant constraints according to the processing constraint characteristics of the flexible job shop;

[0087] S3: Based on the personalized improved NSGA-Ⅱ algorithm module, multi-strategy initialization and dual-strategy selection mechanism are used to optimize the solution quality. Through data normalization technology, the search bias caused by the difference in target magnitude is eliminated, and personalized settings for enterprise needs are added to screen the target population, including the following steps:

[0088] 1-1) Double-layer real number coding is adopted, including "process sorting" and "machine selection". The chromosome length should be Where n represents the total number of workpieces, n i Indicates the number of processes contained in the workpiece;

[0089] 1-2) Decode the encoding scheme using plug-in greedy decoding. The specific decoding principles are as follows:

[0090] If process O ij is the first process of the workpiece, and O ij For machine M k Select the first process to be processed and process it directly at time zero;

[0091] If process O ij It's Machine M k The first process to be processed is in the previous process O i(j-1) Process the process at the time of completion of processing;

[0092] If not, in the machine M k Find the idle segments between the processing steps in turn [ts k ,te k ], calculation process O ij Earliest start time of processing ijk , determine whether the idle time can meet the processing time t required by the process ijk , and satisfy the constraints, process O ij Insert this time period; if the previous process O i(j-1) The processing completion time is less than ts k , process O ij The start processing time is ts k , otherwise the starting processing time is process O i(j-1) The processing completion time;

[0093] If the idle time does not meet the process O ij Processing conditions, then continue to look for the next idle segment [ts k+1 ,te k+1 ];

[0094] 1-3) In order to obtain a better initial solution, multiple strategies are used to initialize the population, including random initialization RS, global minimum machine energy consumption GES, global minimum processing time GS, and local minimum processing time LS. The population is initialized according to the proportion of 20%, 30%, 30%, and 20%;

[0095] 1-4) Use elite selection and binary league to select individuals to be crossed, and design random number selection:

[0096] In the early stage of iteration, if it is less than this random number, two individuals are selected from the parent population and the champion pool of the binary league; if it is greater than this random number, two individuals are selected from the parent population and the champion pool of the elite pool;

[0097] In the later stage of iteration, if it is less than this random number, two individuals are selected from the parent population and the elite pool; if it is greater than this random number, two individuals are selected from the elite pool and the champion pool of the binary league;

[0098] 1-5) Design different crossover and mutation operators:

[0099] Process crossover operator: IPOX is used; the job set is randomly divided into two mutually exclusive subsets, and the job numbers and corresponding machine codes of different parent chromosomes are cross-assigned to the two child chromosomes;

[0100] Machine crossover operator: PMX and SBX are used; the PMX algorithm randomly determines a subset to initialize the exchange process, then screens the job numbers belonging to the subset, and only exchanges the machine codes that meet the conditions, thereby achieving genetic recombination of some characteristics between parent chromosomes; the SBX algorithm initially defines the job subset and generates a control parameter β to regulate the crossover process, and mixes the parent chromosome fragments of the jobs in the subset according to the formula to form a new offspring, where p1 ms 、p2 ms Respectively represent the two parent machine number parts;

[0101] p1 ms =0.5×((1+β)×p1 ms +(1-β)×p2 ms );

[0102] p2 ms =0.5×((1+β)×p2 ms +(1-β)×p1 ms );

[0103] Process mutation operator: exchange mutation and polygon mutation are used; exchange mutation is achieved by exchanging the positions of the job number and its corresponding machine code in the chromosome; polygon mutation uses a random number generation strategy to determine the mutation probability of each gene in the chromosome. When the random number is lower than the predetermined mutation probability threshold, another position on the chromosome is randomly selected for gene exchange;

[0104] Machine mutation operator: single-point mutation is used; a gene site in the chromosome is randomly selected, and the process is determined according to the corresponding gene code; a machine is randomly selected from the list of optional machines in the process to achieve mutation update of the corresponding machine code;

[0105] 1-6) Convert the target importance into optimization priority as the initial input before scheduling. The subsequent steps are as follows:

[0106] Step 1: For each objective function f x Data normalization is performed to eliminate the impact of different dimensions and value ranges, ensuring that the algorithm does not favor targets with larger values ​​during the search process. The processing formula is as follows;

[0107]

[0108] where g x is the function value after normalization, min(f) and max(f) represent the minimum and maximum values ​​of the objective function respectively;

[0109] Step 2: Pareto stratify the population according to the three objective function values. In each stratum, individuals will be sorted according to the specific needs of the enterprise, and those solutions with high priority and low objective function values ​​will be selected first to ensure that they can be selected into the new population first. If the objective function values ​​are the same, the crowding distance will be used to screen the population to avoid the aggregation of solutions.

[0110] Step 3: To prevent only the low non-dominated level from being selected, select the non-dominated individuals in proportion to enter the offspring population, while retaining the elite solution and maintaining the diversity of the population. The selection ratio is calculated as follows:

[0111] num v =min(sp×rank v ,pop size -pop now );

[0112] Among them, num v Indicates the number of individuals selected in the current non-dominated layer, rank v Indicates the number of individuals in the current layer, sp is the selection percentage, pop size is the population size, pop now The current number of children.

[0113] S4: In the DQN-based adaptive decision-making algorithm module, the static features of the workshop and the dynamic features of the population are extracted to construct the state space of the agent. The deep Q network experience replay mechanism is used to train the agent to capture the relationship between scheduling information and scheduling goals for adaptive action decision-making, including the following steps:

[0114] 2-1) Extract static workshop features and dynamic population features from the iteratively generated population information, where the static features include the total number of workpieces N_num, the total number of equipment M_num, the total number of processes for all workpieces N_num, and the number of processes for all workpieces N_num. i _total;

[0115] The following six dynamic features are extracted:

[0116] Uniformity: describes the uniform distribution of the current Pareto frontier solution of the population. The specific calculation is as follows:

[0117]

[0118] Among them, S describes uniformity, |A| represents the number of Pareto frontier solutions, d represents the average level of the current Pareto frontier solution distance, and d v is the solution distance between the vth individual and other individuals, L represents the number of objective functions, f l (x v ) and f l (x u) represent the lth objective function values ​​of the vth individual and the uth individual respectively;

[0119] Solution quality GD: describes the degree of proximity of all individuals in the population to the current Pareto optimal frontier. The specific definition formula is as follows:

[0120]

[0121] Among them, pop represents the population size, ||·||2 represents the L2 norm of the Euclidean distance, and p v .f(x) and pareto.f(x) represent the fitness vectors of the individual and the optimal frontier individual respectively;

[0122] Fitness: the average of the three objectives of the lowest-ranked individual in the non-dominated layer of the current population, and the fitness value of the best individual;

[0123] The action space is constructed by dividing the crossover and mutation parameters with a certain step length. The crossover parameters are divided and generated between 0.5 and 0.9, and the mutation parameters are divided and generated between 0.1 and 0.4. The divided parameters are combined and arranged to generate the action space. The agent selects actions after network training to determine the key parameters P in the current iteration. c , P m ;

[0124] 2-2) The reward function is designed by weighting the impact of different input values ​​in the state space, taking into account the changes in multiple key indicators, giving significant rewards for improved behavior and imposing penalties for declining indicators. The reward weighted calculation is as follows:

[0125] R=ω1R1+ω2R2+…+ω n R n ;

[0126] Among them, ω n Corresponding to different state weights, R n Corresponding to the difference of different state changes;

[0127] When the key indicators decrease at the same time, it is considered that the agent may have approached or reached the optimal solution. At this time, an additional significant reward is given and the termination state d is set. s =1;

[0128] 2-3) Adopt ε-greedy strategy for action selection, batch action selection mechanism. In the exploration phase, the agent randomly selects a batch of actions from the action set. In the utilization phase, the agent selects the top 6 actions with the highest Q value according to the Q value output by the model. ε represents the greedy rate, and the initial value is set to 0.9. The strategy selection is as follows:

[0129]

[0130] The exponential decay is used to adjust the exploration rate and control the decay speed of ε. The specific calculation is as follows:

[0131] ε=min_ε+(max_ε-min_ε)*e -LAMBDA×step ;

[0132] Among them, step is the training time step, LAMBDA is the decay rate, min_ε and max_ε represent the minimum and maximum greedy rates respectively;

[0133] 2-4) The DQN algorithm is used for model training, which includes two networks with the same structure, the Q value network and the Target Q network, which are uniformly constructed as a three-layer fully connected neural network. The ReLU function is used as the activation function to introduce nonlinear characteristics to the network. The specific process is as follows:

[0134] Step 1: Initialize the Q value network and Target Q network parameters, and define the experience buffer to store the state transition tuple Q(s t ,a t ,r t ,s t+1 ,d t ), including the state s t 、Action a t , Reward t , next state s t+1 And the completion mark d t , using fixed parameters to guide population crossover and mutation, this process randomly selects actions in batches, collects experience information and stores it in the experience memory bank, providing a sample basis for subsequent network training;

[0135] Step 2: Randomly extract a batch of data from the experience memory library as training samples, use the behavior value network to predict the Q(s, a) of all actions in the current state, and use the Target Q network to predict the different Q values ​​of the next state, and select the maximum Q value corresponding to each state;

[0136] Step 3: Calculate the target Q value by combining the immediate reward and the future reward. The calculation method is as follows:

[0137] Q(s,a)←Q(s,a)+α(R t+1 +γmax a' Q(s',a')(1-d s ));

[0138] Among them, γ is the discount factor that controls the importance of immediate rewards and experience rewards, α is the learning rate that controls the trade-off between new and old estimates, and d sIndicates whether the current state is a terminal state. For a terminal state, the target Q value is set to 0 to reflect that no further exploration should be done in the terminal state.

[0139] Step 4: To stabilize the learning process and reduce instability during training, the network parameters θ' are soft-updated. In each training, the parameters of the Q value network are moved to the parameters of the Target Q network according to the time step to smooth the learning process:

[0140] t new =t old ×(1-TAU)+e old ×TAU;

[0141] Among them, t old ,t new is the Target Q network parameter before and after the update, e old is the Q value network parameter, and TAU is a hyperparameter that controls the update speed.

[0142] S5: Return the best action predicted by the DQN model to NSGA-II, adjust the optimal parameter ratio according to the real-time status, and improve the quality of multi-objective solution.

[0143] Based on the above steps, the present invention effectively solves the problem that the traditional workshop scheduling method is committed to improving production efficiency, but ignores the requirements for environmental sustainability and the response to the personalized customization needs of enterprises, and proposes an intelligent job shop scheduling method, which includes an improved NSGA-Ⅱ algorithm module based on personalization and an adaptive decision algorithm module based on DQN. Among them, the improved NSGA-Ⅱ algorithm module based on personalization considers efficiency and energy consumption targets, adopts multiple strategies for population initialization, introduces an evolutionary mechanism based on the individual needs of enterprises to screen populations, and continuously screens out approximate optimal solutions through operations such as crossover and mutation of genes. Based on the DQN adaptive decision algorithm module, the state space is constructed by extracting population and workshop characteristics, and the intelligent agent is trained using empirical information. The relationship between the population state and the scheduling target is captured to make action decisions, the optimal proportion parameters are dynamically adjusted, and the space is explored for better solutions. Under the premise of meeting the individual preference needs of enterprises, the efficiency target and the sustainability target are optimized to a greater extent.

[0144] Experimental verification of the intelligent job shop scheduling method of the present invention:

[0145] 1. Test environment

[0146] Windows 11 system, AMD Ryzen 77735HS with Radeon Graphics 3.20GHz, development environment is Python 3.9.10, and deep learning framework is PyTorch 2.3.0.

[0147] 2. Test verification

[0148] Experimental results and analysis are performed on the public benchmarks Brandimarte (MK series), Hurink (ABZ, LA, ORB), and GE.

[0149] (1) Dataset description

[0150] The experiment uses 24 sets of examples from the public benchmarks Brandimarte (MK series), Hurink (ABZ, LA, ORB), and GE for experimental verification. The Brandimarte dataset (MK series) has a workpiece scale of 10 to 20, a process scale of 5 to 15, and a device scale of 4 to 15. The Hurink dataset includes different instance sets such as ABZ, LA, and ORB. The GE dataset contains examples of different scales, with a workpiece scale of 12 to 300 and a device scale of 5 to 20. Since this paper considers the sustainability requirements of Industry 5.0, and the above examples do not have energy consumption indicators, this paper's test examples add processing energy consumption and no-load energy consumption of new equipment on this basis, as shown in Table 1, to meet the consideration of environmental factors.

[0151] Table 1 Equipment energy consumption information

[0152]

[0153] (2) Implementation details and evaluation indicators

[0154] Two typical enterprise demand scenarios are set. In demand one, while maintaining the optimization of workshop production energy consumption and machine load, production efficiency is significantly improved; in demand two, while maintaining production efficiency and machine load optimization, energy consumption is significantly reduced; both scenarios take into account the dual pursuit of high efficiency and sustainability in the manufacturing industry under the background of Industry 5.0.

[0155] Compared with other methods, the enterprise demand is set to reduce the energy consumption of workshop production and the total equipment load while achieving the highest production efficiency; the unified population size is 200, the number of iterations is 100, the initial crossover probability is 0.8, and the mutation probability is 0.2; the DQN model contains three fully connected layers, the initial greedy rate is set to 0.9, the batch size of each training round is 32, and the Adam (adaptive moment estimation) optimizer is used for each training round. The training rounds are 500, and for all examples, the algorithm is repeated 5 times independently to obtain the optimal value.

[0156] In order to evaluate the optimization performance of the model, the completion time (Makespan), total equipment load (Load), total energy consumption of the workshop (Energy) and hypervolume index (HV) are used as evaluation criteria, which are defined as follows:

[0157] Makespan = f1(i);

[0158] Load = f2(i);

[0159] Energy = f3(i);

[0160]

[0161] Among them, f1(i), f1(i), f1(i) correspond to the three fitness values ​​of the individual respectively, R3 represents the maximum value in the function f3(i), and f1(prev) is the first target value of the previous individual (in the sorted list).

[0162] (3) Analysis of comparative experimental results

[0163] In order to verify the adaptive adjustment ability in different cases, a basic NSGA-Ⅱ model without adaptive adjustment ability is set for comparison. The experimental results are as follows: Fig. 9 As shown in the figure, the iteration effect of the target value under 4 groups of examples is shown. The mean values ​​of the target functions of different cases have decreased significantly. Compared with the basic NSGA-Ⅱ model without adaptive adjustment capability, the method in this paper significantly improves the quality of the initial solution. This is because the consideration of the workshop efficiency and energy consumption target requirements is added when the initial solution is generated. The DQN-NSGA-Ⅱ model with adaptive capability shows excellent performance in the overall optimization effect. This is because the historical information in the optimization process is used to enable the model to adaptively adjust the crossover and mutation probabilities during the iteration process, and obtain more reasonable parameters to ensure the solution quality, realizing the further optimization of efficiency and energy consumption targets under the vision of sustainability of Industry 5.0.

[0164] Furthermore, in order to illustrate the superiority of DQN-NSGA-Ⅱ in solving FJSP under the requirements of Industry 5.0, two scheduling optimization methods are selected for comparison with the method in this paper. The MODPSO algorithm is selected from the meta-heuristic algorithm, and the improved Q-learning algorithm is selected from the reinforcement learning. The enterprise demand is set to reduce the production energy consumption of the workshop and the total load of the equipment while achieving the highest production efficiency; the unified population size is 200, the number of iterations is 100, the initial crossover probability is 0.8, and the mutation probability is 0.2; Makespan, Load and Enery in Table 2 represent the maximum completion time, the total load of the equipment and the total energy consumption of the workshop, respectively. For all examples, the three algorithms are repeated 5 times independently to obtain the optimal value.

[0165] Table 2 Benchmark example solution results

[0166]

[0167]

[0168] The bold in Table 2 indicates the optimal values ​​of the indicators in the three algorithms. It can be clearly seen from the solution results that the completion time of the DQN-NSGA-Ⅱ algorithm is lower than that of the other two algorithms in all cases, the energy consumption is lower than that of the other two algorithms in 95% of the cases, and the equipment load is lower than that of the other two algorithms in 85% of the cases. Among them, the Q-learning algorithm weights multiple objective functions into a single objective for optimization. Due to the large difference in the magnitude of the indicators, the optimization is unbalanced, and the energy consumption increases significantly while reducing the completion time; MODPSO cannot be dynamically adjusted due to fixed parameters, and the model has poor adaptability, resulting in differences in optimization performance on cases of different scales. This paper adopts multi-strategy initialization to obtain better initial solution quality under the same number of iterations. Through dual-strategy selection, the exploration and development before and after the iteration are balanced, and the personalized evolution mechanism considering the needs of the enterprise is introduced to customize the target population for screening; DQN is used to train the intelligent agent, and the population crossover and mutation ratio is intelligently adjusted according to the current scheduling status and historical data to adapt to workpieces and equipment of different scales, which can better consider environmental sustainability while maintaining high efficiency. The overall algorithm has been significantly improved in terms of optimization performance, effectively improving production efficiency and reducing production energy consumption.

[0169] In order to more intuitively compare the distribution of Pareto solutions obtained by the three methods, HV is used to measure the size of the Pareto frontier solution set covering the target space. The comparison results are shown in Fig.10 As shown in the figure, the HV mean obtained by algorithm DQN-NSGA-Ⅱ is better than the other two algorithms in all examples, indicating that the distribution of Pareto solution set is more uniform and there are more scheduling schemes for enterprises to choose from.

[0170] In summary, the intelligent job shop scheduling method proposed in this paper has more advantages than the metaheuristic and reinforcement learning single model algorithms. By considering the optimization objectives of the workshop and the individual needs of the enterprise, this method effectively utilizes the static and population dynamic experience information of the workshop in the iterative process, and provides an effective solution for flexible job shop scheduling from the perspective of Industry 5.0.

[0171] 3. Conclusion

[0172] In view of the fact that Industry 5.0 emphasizes meeting the core needs of human beings while respecting the environment and ecology to maintain its sustainability, the scheduling of job shops should take into account the sustainability and personalized needs of enterprises while pursuing production efficiency. This study proposes an intelligent job shop scheduling method. First, considering the production efficiency and sustainability needs comprehensively, three performance indicators are determined as optimization targets; secondly, a personalized multi-strategy improved NSGA-Ⅱ is proposed, which selects the target population by considering the personalized evolution mechanism of enterprise needs and explores the scheduling solution that meets the optimization needs of enterprises; the DQN network model is used to train the intelligent agent, capture the relationship between population information and scheduling goals for dynamic decision-making, predict the best action, improve the solution quality on different examples, and maximize the improvement of production efficiency and reduction of energy consumption; finally, the scheduling ability of the algorithm under the personalized needs of enterprises is verified through experiments. Compared with other scheduling algorithms, the values ​​of various objectives are effectively optimized on different examples, and personalized and environmentally friendly factors are comprehensively considered, which provides support for realizing the personalized and sustainable vision in Industry 5.0.

Claims

1. An intelligent job shop scheduling method, characterized in that: The steps include: S1: Build an improved NSGA-Ⅱ algorithm module based on personalization and an adaptive decision algorithm module based on DQN; S2: Establish to minimize the maximum completion time C max 、Total equipment load L all And the total energy consumption of the workshop E all To optimize the book order model of the target, design relevant constraints according to the processing constraint characteristics of the flexible job shop; S3: Based on the personalized improved NSGA-Ⅱ algorithm module, multi-strategy initialization and dual-strategy selection mechanism are used to optimize the solution quality. Through data normalization technology, the search bias caused by the difference in target magnitude is eliminated, and personalized settings for enterprise needs are added to screen the target population; S4: In the DQN-based adaptive adjustment of key parameters module, the static characteristics of the workshop and the dynamic characteristics of the population are extracted to construct the state space of the intelligent agent. The deep Q network experience replay mechanism is used to train the intelligent agent to capture the relationship between scheduling information and scheduling goals for adaptive action decision-making; S5: Return the best action predicted by the DQN model to NSGA-II, adjust the optimal parameter ratio according to the real-time status, and improve the quality of multi-objective solution.

2. The intelligent job shop scheduling method according to claim 1, characterized in that: In step S3, a multi-strategy initialization and dual-strategy selection mechanism is used to optimize the solution quality. The search bias caused by the difference in target magnitude is eliminated through data normalization technology to avoid the optimization imbalance caused by the difference in numerical scale. Then, personalized settings for user needs are added to screen the target population, including the following steps: 1-1) Double-layer real number coding is adopted, including "process sorting" and "machine selection". The chromosome length should be Where n represents the total number of workpieces, n i Indicates the number of processes contained in the workpiece; 1-2) Decode the encoding scheme using plug-in greedy decoding. The specific decoding principles are as follows: If process O ij is the first process of the workpiece, and O ij For machine M k Select the first process to be processed and process it directly at time zero; If process O ij It's Machine M k The first process to be processed is in the previous process O i(j-1) Process the process at the time of completion of processing; If not, in the machine M k Find the idle segments between the processing steps in turn [ts k ,te k ], calculation process O ij Earliest start time of processing ijk , determine whether the idle time can meet the processing time t required by the process ijk , and satisfy the constraints, process O ij Insert this time period; if the previous process O i(j-1) The processing completion time is less than ts k , process O ij The start processing time is ts k , otherwise the starting processing time is process O i(j-1) The processing completion time; If the idle time does not meet the process O ij Processing conditions, then continue to look for the next idle segment [ts k+1 ,te k+1 ]; 1-3) In order to obtain a better initial solution, multiple strategies are used to initialize the population, including random initialization RS, global minimum machine energy consumption GES, global minimum processing time GS, and local minimum processing time LS. The population is initialized according to the proportion of 20%, 30%, 30%, and 20%; 1-4) Use elite selection and binary league to select individuals to be crossed, and design random number selection: In the early stage of iteration, if it is less than this random number, two individuals are selected from the parent population and the champion pool of the binary league; if it is greater than this random number, two individuals are selected from the parent population and the champion pool of the elite pool; In the later stage of iteration, if it is less than this random number, two individuals are selected from the parent population and the elite pool; if it is greater than this random number, two individuals are selected from the elite pool and the champion pool of the binary league; 1-5) Design different crossover and mutation operators: Process crossover operator: IPOX is used; the job set is randomly divided into two mutually exclusive subsets, and the job numbers and corresponding machine codes of different parent chromosomes are cross-assigned to the two child chromosomes; Machine crossover operator: PMX and SBX are used; the PMX algorithm randomly determines a subset to initialize the exchange process, then screens the job numbers belonging to the subset, and only exchanges the machine codes that meet the conditions, thereby achieving genetic recombination of some characteristics between parent chromosomes; the SBX algorithm initially defines the job subset and generates a control parameter β to regulate the crossover process, and mixes the parent chromosome fragments of the jobs in the subset according to the formula to form a new offspring, where p1 ms 、p2 ms Respectively represent the two parent machine number parts; p1 ms =0.5×((1+β)×p1 ms +(1-β)×p2 ms ); p2 ms =0.5×((1+β)×p2 ms +(1-β)×p1 ms ); Process mutation operator: exchange mutation and polygon mutation are used; exchange mutation is achieved by exchanging the positions of the job number and its corresponding machine code in the chromosome; polygon mutation uses a random number generation strategy to determine the mutation probability of each gene in the chromosome. When the random number is lower than the predetermined mutation probability threshold, another position on the chromosome is randomly selected for gene exchange; Machine mutation operator: single-point mutation is used; a gene site in the chromosome is randomly selected, and the process is determined according to the corresponding gene code; a machine is randomly selected from the list of optional machines in the process to achieve mutation update of the corresponding machine code; 1-6) Convert the target importance into optimization priority as the initial input before scheduling. The subsequent steps are as follows: Step 1: For each objective function f x Data normalization is performed to eliminate the impact of different dimensions and value ranges, ensuring that the algorithm does not favor targets with larger values ​​during the search process. The processing formula is as follows; where g x is the function value after normalization, min(f) and max(f) represent the minimum and maximum values ​​of the objective function respectively; Step 2: Pareto stratify the population according to the three objective function values. In each stratum, individuals will be sorted according to the specific needs of the enterprise, and those solutions with high priority and low objective function values ​​will be selected first to ensure that they can be selected into the new population first. If the objective function values ​​are the same, the crowding distance will be used to screen the population to avoid the aggregation of solutions. Step 3: To prevent only the low non-dominated level from being selected, select the non-dominated individuals in proportion to enter the offspring population, while retaining the elite solution and maintaining the diversity of the population. The selection ratio is calculated as follows: num v =min(sp×rank v ,pop size -pop now ); Among them, num v Indicates the number of individuals selected in the current non-dominated layer, rank v Indicates the number of individuals in the current layer, sp is the selection percentage, pop size is the population size, pop now The current number of children.

3. The intelligent job shop scheduling method according to claim 1, characterized in that: In step S4, in the DQN adaptive decision algorithm module, the static characteristics of the workshop and the dynamic characteristics of the population are extracted to construct the state space of the agent, and the deep Q network experience playback mechanism is used to train the agent to capture the relationship between the scheduling information and the scheduling target to make adaptive action decisions, including the following steps: 2-1) Extract static workshop features and dynamic population features from the iteratively generated population information, where the static features include the total number of workpieces N_num, the total number of equipment M_num, the total number of processes for all workpieces N_num, and the number of processes for all workpieces N_num. i _total; The following six dynamic features are extracted: Uniformity: describes the uniform distribution of the current Pareto frontier solution of the population. The specific calculation is as follows: Among them, S describes uniformity, |A| represents the number of Pareto frontier solutions, Indicates the average level of the current Pareto frontier solution distance, d v is the solution distance between the vth individual and other individuals, L represents the number of objective functions, f l (x v ) and f l (x u ) represent the lth objective function values ​​of the vth individual and the uth individual respectively; Solution quality GD: describes the degree of proximity of all individuals in the population to the current Pareto optimal frontier. The specific definition formula is as follows: Among them, pop represents the population size, ||·||2 represents the L2 norm of the Euclidean distance, and p v .f(x) and pareto.f(x) represent the fitness vectors of the individual and the optimal frontier individual respectively; Fitness: the average of the three objectives of the lowest-ranked individual in the non-dominated layer of the current population, and the fitness value of the best individual; The action space is constructed by dividing the crossover and mutation parameters with a certain step length. The crossover parameters are divided and generated between 0.5 and 0.9, and the mutation parameters are divided and generated between 0.1 and 0.

4. The divided parameters are combined and arranged to generate the action space. The agent selects actions after network training to determine the key parameters P in the current iteration. c , P m ; 2-2) The reward function is designed by weighting the impact of different input values ​​in the state space, taking into account the changes in multiple key indicators, giving significant rewards for improved behavior and imposing penalties for declining indicators. The reward weighted calculation is as follows: R=ω1R1+ω2R2+…+ω n R n ; Among them, ω n Corresponding to different state weights, R n Corresponding to the difference of different state changes; When the key indicators decrease at the same time, it is considered that the agent may have approached or reached the optimal solution. At this time, an additional significant reward is given and the termination state d is set. s =1; 2-3) Adopt ε-greedy strategy for action selection, batch action selection mechanism. In the exploration phase, the agent randomly selects a batch of actions from the action set. In the utilization phase, the agent selects the top 6 actions with the highest Q value according to the Q value output by the model. ε represents the greedy rate, and the initial value is set to 0.

9. The strategy selection is as follows: The exponential decay is used to adjust the exploration rate and control the decay speed of ε. The specific calculation is as follows: ε=min_ε+(max_ε-min_ε)*e -LAMBDA×step ; Among them, step is the training time step, LAMBDA is the decay rate, min_ε and max_ε represent the minimum and maximum greedy rates respectively; 2-4) The DQN algorithm is used for model training, which includes two networks with the same structure, the Q value network and the Target Q network, which are uniformly constructed as a three-layer fully connected neural network. The ReLU function is used as the activation function to introduce nonlinear characteristics to the network. The specific process is as follows: Step 1: Initialize the Q value network and Target Q network parameters, and define the experience buffer to store the state transition tuple Q(s t ,a t ,r t ,s t+1 ,d t ), including the state s t 、Action a t , Reward t , next state s t+1 And the completion mark d t , using fixed parameters to guide population crossover and mutation, this process randomly selects actions in batches, collects experience information and stores it in the experience memory bank, providing a sample basis for subsequent network training; Step 2: Randomly extract a batch of data from the experience memory library as training samples, use the behavior value network to predict the Q(s, a) of all actions in the current state, and use the Target Q network to predict the different Q values ​​of the next state, and select the maximum Q value corresponding to each state; Step 3: Calculate the target Q value by combining the immediate reward and the future reward. The calculation method is as follows: Q(s,a)←Q(s,a)+α(R t+1 +γmax a' Q(s',a')(1-d s )); Among them, γ is the discount factor that controls the importance of immediate rewards and experience rewards, α is the learning rate that controls the trade-off between new and old estimates, and d s Indicates whether the current state is a terminal state. For a terminal state, the target Q value is set to 0 to reflect that no further exploration should be done in the terminal state. Step 4: To stabilize the learning process and reduce instability during training, the network parameters θ' are soft-updated. In each training, the parameters of the Q value network are moved to the parameters of the Target Q network according to the time step to smooth the learning process: t new =t old ×(1-YEAR)+e old × YEAR; Among them, t old ,t new is the Target Q network parameter before and after the update, e old is the Q value network parameter, and TAU is a hyperparameter that controls the update speed.

Images