Two-stage deep reinforcement learning method suitable for multi-target dynamic FJSP
Through the two-stage deep reinforcement learning method, the reward function and scheduling behavior are calculated using the dual-layer neural network, and the problem of solving too long in the multi-objective dynamic flexible operation workshop scheduling problem is solved, achieving more efficient scheduling optimization and model convergence.
Patent Information
- Application Number
- CN202411144716.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-12
AI Technical Summary
The existing scheduling optimization methods have been solved in the multi-objective dynamic flexible operation workshop scheduling problem, making it difficult to respond to random events in the production workshop.
Using a two-stage deep reinforcement learning method, a two-layer neural network is used to calculate reward functions and scheduling behaviors, and the network parameters are updated through external non-dominant sets, which shortens the network parameter training time and improves the convergence of the model.
This method can effectively solve multi-objective problems, shorten the solution time, improve the convergence and optimization capabilities of the model, and significantly improve the scheduling efficiency of the production workshop.
Smart Images

Figure CN118938682A_ABST
Abstract
Description
Technical Field
[0001] The present invention is applicable to the technical field of scheduling optimization, and in particular to a two-stage deep reinforcement learning method applicable to multi-objective dynamic FJSP. Background Art
[0002] At present, with the increasing demand of manufacturing industry, production workshops are gradually facing a large number of production orders. Due to the large number of orders and the flexibility of workshop production, it is difficult for traditional production scheduling rules to find a satisfactory solution within the specified delivery time; in addition, there are many uncertain times in the workshop production process, such as machine failure, new workpiece insertion, etc. Such random events will also reduce production efficiency and even cause chaos in production.
[0003] With the continuous development of intelligent manufacturing technology, when facing such random events, workshops need to have short response time and optimal rescheduling strategy. Generally, the workshop scheduling problem for solving multiple optimization objectives is called Dynamic Multi-objective Flexible Job Shop Problem (DMO-FJSP). However, the existing methods such as swarm intelligence algorithms take too long to solve scheduling optimization and are difficult to meet such requirements. Summary of the invention
[0004] The present invention provides a two-stage deep reinforcement learning method suitable for multi-objective dynamic FJSP, aiming to solve the problem that the solution time of existing scheduling optimization methods is too long.
[0005] In order to solve the above technical problems, the present invention provides a two-stage deep reinforcement learning method suitable for multi-objective dynamic FJSP, comprising the following steps:
[0006] S1. Set an external non-dominated set to obtain real-time processing data of the workshop;
[0007] S2. Calculating a plurality of workpiece state values based on the real-time processing data of the workshop, and combining the different types of workpiece state values to obtain a state vector;
[0008] S3, inputting the state vector into the preset first-stage double-layer network to calculate the reward function calculation method of the current round;
[0009] S4, inputting the state vector into a preset second-stage double-layer network to calculate the scheduling behavior of the current round;
[0010] S5, execute the scheduling behavior to make the workshop enter a new execution state, calculate the second state vector according to the method of steps S2-S4, calculate the reward function according to the difference between the state vector and the second state vector, and store the state vector, the reward function calculation method, the scheduling behavior and the reward function data into the first experience playback pool and the second experience recycling pool according to the stage;
[0011] S6, randomly extracting data from the first experience replay pool, the second experience recycling pool or the external non-dominated set, and updating the first-stage double-layer network and the second-stage double-layer network;
[0012] S7. Input the scheduling behavior into the external non-dominated set, perform non-dominated sorting on the external non-dominated set, retain the non-dominated solutions, and obtain the scheduling result of the current round according to the non-dominated solutions.
[0013] Furthermore, the real-time processing data of the workshop includes the number of processed workpieces, the process of completing the processing of the workpieces, and the processing machines available for each process. The various workpiece status values include the average machine utilization rate U ave (t), standard deviation of machine utilization U std (t), total process completion rate CRO(t), average workpiece completion rate CRJ ave (t), standard deviation of workpiece completion rate CRJ std (t), where:
[0014]
[0015] m represents the number of machines, U k (t) represents the utilization rate of the kth machine, and t represents the time of the current execution scheduling optimization;
[0016]
[0017] n i represents the total number of processes for the i-th job, OP i (t) represents the completed process of the i-th workpiece;
[0018]
[0019] n represents the number of workpieces, CRJ i (t) indicates that the i-th workpiece has completed the process;
[0020]
[0021] Furthermore, the state vector obtained by combining different types of workpiece state values is defined as state, the preset first-stage two-layer network includes a first-order one-layer network and a first-order two-layer network, the output node of the first-stage two-layer network is 2, and step S3 includes the following sub-steps:
[0022] Input the state vector state into the first-order one-layer network to obtain the reward function calculation method goal and the corresponding first index value index1;
[0023] Input the state vector state into the first-order two-layer network, and select the corresponding network calculation value Q according to the index value index to calculate the first label value It satisfies:
[0024]
[0025] Wherein, s' represents the state vector state, a1' represents the first index value index1, r represents the preset reward value, γ represents the preset discount coefficient, Q w , Q w -represent the processing process of the first-order layer 1 network and the first-order layer 2 network respectively.
[0026] Furthermore, the preset second-stage two-layer network includes a second-order one-layer network and a second-order two-layer network, the output node of the second-stage two-layer network is 4, and step S4 includes the following sub-steps:
[0027] Input the state vector state into the second-order one-layer network to obtain the scheduling action action and the corresponding second index value index2;
[0028] The state vector state is input into the second-order two-layer network, and the corresponding network calculation value Q is selected according to the second index value index to calculate the second label value It satisfies:
[0029]
[0030] Wherein, a2' represents the second index value index2.
[0031] Furthermore, the second state vector is defined as state_, the reward function is reward, the first experience replay pool is replay1, and the second experience replay pool is replay2. In step S5, the state vector state, the reward function calculation method goal, the reward function reward and the second state vector state_ are put into the first experience replay pool, and the state vector state, the scheduling behavior action, the reward function reward and the second state vector state_ are put into the second experience replay pool.
[0032] Furthermore, step S6 includes the following sub-steps:
[0033] In a random manner, extracting data of a preset size from the first experience replay pool replay1, the second experience recycling pool replay2, and the external non-dominated set, and sending the data to the first-order one-layer network and the second-order one-layer network for parameter training;
[0034] According to the first label value The second label value Performing gradient descent with the network calculation value Q calculated based on the state vector state;
[0035] Assign the network parameters of the first-order one-layer network and the second-order one-layer network to the first-order two-layer network and the second-order two-layer network respectively to complete the network update
[0036] The beneficial effect achieved by the present invention is that a two-stage deep reinforcement learning method suitable for multi-objective dynamic FJSP is proposed. The method uses a two-layer neural network to solve the reward function calculation method to be used in the current round, and uses another two-layer neural network in a staged manner to solve the scheduling decision to be executed under the premise of satisfying this reward function, and saves the final scheduling result to an external non-dominated set, and uses the solution in the non-dominated set to update the network parameters. Compared with the existing reinforcement learning algorithm, this process can solve multi-objective problems, and at the same time greatly shortens the network parameter training time and increases the convergence of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is a schematic diagram of the steps of a two-stage deep reinforcement learning method applicable to multi-objective dynamic FJSP provided by an embodiment of the present invention;
[0038] Figure 2 It is an overall logic schematic diagram for regulating the workpiece process in a production workshop provided by an embodiment of the present invention;
[0039] Figure 3 It is a process logic diagram of a two-stage deep reinforcement learning method applicable to multi-objective dynamic FJSP provided by an embodiment of the present invention;
[0040] Figure 4 It is a schematic diagram of the calculation time comparison result between the TS-DDQN method provided by an embodiment of the present invention and the existing scheduling method;
[0041] Figure 5 It is a rain cloud diagram of the GD and IGD indicators showing the comparison results of the solution quality of the TS-DDQN method provided by the embodiment of the present invention and the existing reinforcement learning algorithm. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0043] Please refer to Figure 1 , Figure 1 1 is a schematic flow chart of the steps of a two-stage deep reinforcement learning method for multi-objective dynamic FJSP provided by an embodiment of the present invention. The two-stage deep reinforcement learning method for multi-objective dynamic FJSP includes the following steps:
[0044] S1. Set an external non-dominated set to obtain real-time processing data of the workshop;
[0045] S2. Calculating a plurality of workpiece state values based on the real-time processing data of the workshop, and combining the different types of workpiece state values to obtain a state vector;
[0046] S3, inputting the state vector into the preset first-stage double-layer network to calculate the reward function calculation method of the current round;
[0047] S4, inputting the state vector into a preset second-stage double-layer network to calculate the scheduling behavior of the current round;
[0048] S5, execute the scheduling behavior to make the workshop enter a new execution state, calculate the second state vector according to the method of steps S2-S4, calculate the reward function according to the difference between the state vector and the second state vector, and store the state vector, the reward function calculation method, the scheduling behavior and the reward function data into the first experience playback pool and the second experience recycling pool according to the stage;
[0049] S6, randomly extracting data from the first experience replay pool, the second experience recycling pool or the external non-dominated set, and updating the first-stage double-layer network and the second-stage double-layer network;
[0050] S7. Input the scheduling behavior into the external non-dominated set, perform non-dominated sorting on the external non-dominated set, retain the non-dominated solutions, and obtain the scheduling result of the current round according to the non-dominated solutions.
[0051] Specifically, in the embodiment of the present invention, the experience recycling pool and the external non-dominated set are used to optimize the training process of the double-layer network.
[0052] The experience replay pool refers to the training data that the model will continuously generate during operation, such as state, behavior, reward, next moment state, etc., by collecting these data and storing them in a capacity pool. During the training process, data is taken from the experience replay pool with a certain probability as the input and label required for network training. In the embodiment of the present invention, since the data in the experience replay pool is extracted in random batches with a certain probability, the data is random, so it can reduce data correlation, improve data utilization, enhance learning stability, and avoid local optimality.
[0053] The external non-dominated set refers to the existence of a set in which each element stored in the set does not dominate each other. The relationship between dominated, non-dominated, and non-dominated solutions is explained as follows: In multi-objective optimization problems, "dominated" and "non-dominated" are two key concepts, which are mainly used to describe the relative advantages and disadvantages between solutions. When faced with multiple objective functions, it is difficult to find a solution that can maximize or minimize all objectives at the same time because there may be conflicts between the objectives; therefore, the goal of optimization is usually to find a set of solutions that are as good as possible among all objectives. This is the so-called "non-dominated solution" or "Pareto optimal solution";
[0054] The concept of "dominance" means that in multi-objective optimization, for any two solutions S1 and S2, if S1 is not worse than S2 in all objectives and is better than S2 in at least one objective, S1 can be said to dominate S2;
[0055] The concept of "non-dominated solution" means that there is no solution that dominates it. In other words, for a non-dominated solution S1, there is no other solution S2 that makes S2 no worse than S1 in all objectives and better in at least one objective; in mathematical form, if there is no solution S2 that satisfies the above domination condition, then S1 is non-dominated.
[0056] The "non-dominated set" means that the set of non-dominated solutions forms the Pareto front, which is the set of points consisting of all objective function values in the objective space, where no point can improve one objective without compromising at least one other objective. The Pareto front represents the boundary of the best possible compromise solution in a multi-objective optimization problem.
[0057] In the embodiment of the present invention, the execution of the method begins with the occurrence of an executable schedule in the current workshop, and the overall logic of regulating the workpiece process in the production workshop through scheduling is as follows: Figure 2 shown.
[0058] Furthermore, the real-time processing data of the workshop includes the number of processed workpieces, the process of completing the processing of the workpieces, and the processing machines available for each process. The various workpiece status values include the average machine utilization rate U ave (t), standard deviation of machine utilization U std (t), total process completion rate CRO(t), average workpiece completion rate CRJ ave (t), standard deviation of workpiece completion rate CRJ std (t), where:
[0059]
[0060] m represents the number of machines, U k (t) represents the utilization rate of the kth machine, and t represents the time of the current execution scheduling optimization;
[0061]
[0062] n i represents the total number of processes for the i-th job, OP i (t) represents the completed process of the i-th workpiece;
[0063]
[0064] n represents the number of workpieces, CRJ i (t) indicates that the i-th workpiece has completed the process;
[0065]
[0066] For details, please refer to Figure 3In the method logic diagram shown, there are differences between the double-layer networks in the first stage and the second stage, mainly manifested in the different number of output nodes. There are only 2 output nodes in the double-layer network in the first stage because the output of this stage is mainly to determine the objective function, and there are only 2 objective functions in the embodiment of the present invention; there are 4 output nodes in the double-layer network in the second stage, which is mainly because the double-layer network in the second stage is mainly to determine the scheduling behavior, and the scheduling behavior in the embodiment of the present invention is 4. The double-layer network structures in these two stages have no relationship with each other and are independent of each other. The only connection is that the number of input nodes in these two stages must be the same as the dimension of the state vector (6 dimensions in the embodiment of the present invention). The first layer and the second layer in the double-layer network of each stage are related. The two-layer network structure is the same, and the parameters are copied every certain number of training rounds to save training time. The parameter copying step is reflected in step S6.
[0067] Furthermore, the state vector obtained by combining different types of workpiece state values is defined as state, the preset first-stage two-layer network includes a first-order one-layer network and a first-order two-layer network, the output node of the first-stage two-layer network is 2, and step S3 includes the following sub-steps:
[0068] Input the state vector state into the first-order one-layer network to obtain the reward function calculation method goal and the corresponding first index value index1;
[0069] Input the state vector state into the first-order two-layer network, and select the corresponding network calculation value Q according to the index value index to calculate the first label value It satisfies:
[0070]
[0071] Wherein, s' represents the state vector state, a1' represents the first index value index1, r represents the preset reward value, γ represents the preset discount coefficient, Q w , Q w -represent the processing process of the first-order layer 1 network and the first-order layer 2 network respectively.
[0072] Specifically, in the embodiment of the present invention, the number of nodes output by the first-stage double-layer network is 2, that is, the value of the index is either 0 or 1. 0 indicates that the objective function of this round of training selects the first one, and 1 indicates that the objective function of this round of training selects the second one. The embodiment of the present invention obtains the calculation method of the objective function of this round of training through the index.
[0073] Exemplarily, the preset reward value may be calculated as shown in Table 1 below.
[0074] Table 1 Calculation method of preset reward value
[0075]
[0076]
[0077] Furthermore, the preset second-stage two-layer network includes a second-order one-layer network and a second-order two-layer network, the output node of the second-stage two-layer network is 4, and step S4 includes the following sub-steps:
[0078] Input the state vector state into the second-order one-layer network to obtain the scheduling action action and the corresponding second index value index2;
[0079] The state vector state is input into the second-order two-layer network, and the corresponding network calculation value Q is selected according to the second index value index to calculate the second label value It satisfies:
[0080]
[0081] Wherein, a2' represents the second index value index2.
[0082] Furthermore, the second state vector is defined as state_, the reward function is reward, the first experience replay pool is replay1, and the second experience replay pool is replay2. In step S5, the state vector state, the reward function calculation method goal, the reward function reward and the second state vector state_ are put into the first experience replay pool, and the state vector state, the scheduling behavior action, the reward function reward and the second state vector state_ are put into the second experience replay pool.
[0083] Furthermore, step S6 includes the following sub-steps:
[0084] In a random manner, extracting data of a preset size from the first experience replay pool replay1, the second experience recycling pool replay2, and the external non-dominated set, and sending the data to the first-order one-layer network and the second-order one-layer network for parameter training;
[0085] According to the first label value The second label value Performing gradient descent with the network calculation value Q calculated based on the state vector state;
[0086] The network parameters of the first-order one-layer network and the second-order one-layer network are assigned to the first-order two-layer network and the second-order two-layer network respectively to complete the network update.
[0087] In step S7, non-dominated sorting is used. This method refers to the fact that in multi-objective optimization, there is usually no single optimal solution, but a series of solutions, which may be better than other solutions in some objectives, but worse in other objectives. Such a solution set is called a Pareto optimal solution set, and non-dominated sorting is a method used to identify and organize these Pareto optimal solutions. In layman's terms, a batch of solution sets are stratified into solution sets of different qualities through a sorting method. The solutions on the first layer dominate the solutions on the following layers, but the solutions on the first layer do not dominate each other, and so on. In this way, a batch of solutions can be divided into different levels.
[0088] The embodiment of the present invention uses this method to retain only the first-level solutions in the non-dominated set, that is, the best solutions after screening, so as to enhance the quality of data.
[0089] The non-dominated solution set retained in step S7 is the best solution set selected from a batch of solutions. These solutions are essentially calculated by state values, because these solutions represent the processing sequence of workshop workpieces, and the processing sequence is calculated based on the real-time workshop processing status. Therefore, the quality of this solution can directly reflect the status of the real-time processing environment.
[0090] Specifically, the overall algorithm of the two-stage deep reinforcement learning method applicable to multi-objective dynamic FJSP proposed in the embodiment of the present invention can be implemented with reference to the method in Table 2.
[0091] Table 2 Overall algorithm of the two-stage deep reinforcement learning method for multi-objective dynamic FJSP
[0092]
[0093]
[0094] The two-stage deep reinforcement learning method for multi-objective dynamic FJSP proposed in the embodiment of the present invention is defined as the TS-DDQN method. The embodiment of the present invention also designs a calculation time comparison between the TS-DDQN method and the existing scheduling method, such as Figure 4As shown in the figure, it can be seen that the time difference between the TS-DDQN method and other scheduling rules is not large, which shows that the TS-DDQN method and the scheduling rules are fast enough for DMOFJSP to run. In actual situations, this time will be higher than the calculated result, mainly because the state value of the network is calculated based on the actual workshop production environment, and a small amount of time may be spent in the data collection and transmission process; however, despite the slight increase in calculation time, the TS-DDQN method can still obtain a high-quality Pareto optimal solution, indicating that the TS-DDQN method has advantages in both optimization ability and computational efficiency.
[0095] Figure 5 The GD and IGD indicators rain cloud diagram showing the comparison results of the solution quality of the TS-DDQN method and the existing reinforcement learning algorithms, including the two-level deep Q network (THDQN), the dual deep Q network (DDQN) and the single-stage deep Q network (SSDQN), shows that the TS-DDQN method is superior to other DRL methods. In instances of various sizes, the GD, IGD and distribution indicators of the TS-DDQN method are significantly smaller, while maintaining the stability of the GD and IGD values, highlighting its excellent performance. This robust and consistent performance demonstrates the effectiveness of the TS-DDQN algorithm in solving DMOFJSP; in addition, the positions where the GD and IGD appear more frequently obtained by the TS-DDQN method are relatively close to 0, while the points obtained by other DRL methods are relatively scattered, and they are not concentrated near 0, which shows that the effects of the three existing algorithms in the current indicators are far inferior to the two-stage deep reinforcement learning method for multi-objective dynamic FJSP proposed in the embodiment of the present invention.
[0096] The beneficial effect achieved by the present invention is that a two-stage deep reinforcement learning method suitable for multi-objective dynamic FJSP is proposed. The method uses a two-layer neural network to solve the reward function calculation method to be used in the current round, and uses another two-layer neural network in a staged manner to solve the scheduling decision to be executed under the premise of satisfying this reward function, and saves the final scheduling result to an external non-dominated set, and uses the solution in the non-dominated set to update the network parameters. Compared with the existing reinforcement learning algorithm, this process can solve multi-objective problems, and at the same time greatly shortens the network parameter training time and increases the convergence of the model.
[0097] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0098] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0099] The embodiments of the present invention are described above in conjunction with the accompanying drawings. What is disclosed is only the preferred embodiment of the present invention. However, the present invention is not limited to the above-mentioned specific implementation manner. The above-mentioned specific implementation manner is only illustrative rather than restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms and equivalent changes without departing from the scope of protection of the purpose of the present invention and the claims, all of which are within the protection of the present invention.
Claims
1. A two-stage deep reinforcement learning method for multi-objective dynamic FJSP, characterized in that: The following steps are involved: S1. Set an external non-dominated set to obtain real-time processing data of the workshop; S2. Calculating a plurality of workpiece state values based on the real-time processing data of the workshop, and combining the different types of workpiece state values to obtain a state vector; S3, inputting the state vector into the preset first-stage double-layer network to calculate the reward function calculation method of the current round; S4, inputting the state vector into a preset second-stage double-layer network to calculate the scheduling behavior of the current round; S5, execute the scheduling behavior to make the workshop enter a new execution state, calculate the second state vector according to the method of steps S2-S4, calculate the reward function according to the difference between the state vector and the second state vector, and store the state vector, the reward function calculation method, the scheduling behavior and the reward function data into the first experience playback pool and the second experience recycling pool according to the stage; S6, randomly extracting data from the first experience replay pool, the second experience recycling pool or the external non-dominated set, and updating the first-stage double-layer network and the second-stage double-layer network; S7. Input the scheduling behavior into the external non-dominated set, perform non-dominated sorting on the external non-dominated set, retain the non-dominated solutions, and obtain the scheduling result of the current round according to the non-dominated solutions.
2. The two-stage deep reinforcement learning method for multi-objective dynamic FJSP as claimed in claim 1, characterized in that: The real-time processing data of the workshop includes the number of processed workpieces, the process of completing the processing of the workpieces, and the processing machines available for each process. The various workpiece status values include the average machine utilization rate U ave (t), standard deviation of machine utilization U std (t), total process completion rate CRO(t), average workpiece completion rate CRJ ave (t), standard deviation of workpiece completion rate CRJ std (t), where: m represents the number of machines, U k (t) represents the utilization rate of the kth machine, and t represents the time of the current execution scheduling optimization; n i represents the total number of processes for the i-th job, OP i (t) represents the completed process of the i-th workpiece; n represents the number of workpieces, CRJ i (t) indicates that the i-th workpiece has completed the process; 3. The two-stage deep reinforcement learning method for multi-objective dynamic FJSP as claimed in claim 2, characterized in that: The state vector obtained by combining different types of workpiece state values is defined as state, the preset first-stage double-layer network includes a first-order one-layer network and a first-order two-layer network, the output node of the first-stage double-layer network is 2, and step S3 includes the following sub-steps: Input the state vector state into the first-order one-layer network to obtain the reward function calculation method goal and the corresponding first index value index1; Input the state vector state into the first-order two-layer network, and select the corresponding network calculation value Q according to the index value index to calculate the first label value It satisfies: Wherein, s' represents the state vector state, a1' represents the first index value index1, r represents the preset reward value, γ represents the preset discount coefficient, Q w , Q w- They respectively represent the processing processes of the first-order layer 1 network and the first-order layer 2 network.
4. The two-stage deep reinforcement learning method for multi-objective dynamic FJSP as claimed in claim 3, characterized in that: The preset second-stage two-layer network includes a second-order one-layer network and a second-order two-layer network, the output node of the second-stage two-layer network is 4, and step S4 includes the following sub-steps: Input the state vector state into the second-order one-layer network to obtain the scheduling action action and the corresponding second index value index2; The state vector state is input into the second-order two-layer network, and the corresponding network calculation value Q is selected according to the second index value index to calculate the second label value It satisfies: Wherein, a2' represents the second index value index2.
5. The two-stage deep reinforcement learning method for multi-objective dynamic FJSP as claimed in claim 4, characterized in that: Define the second state vector as state_, the reward function as reward, the first experience replay pool as replay1, and the second experience replay pool as replay2. In step S5, put the state vector state, the reward function calculation method goal, the reward function reward and the second state vector state_ into the first experience replay pool, and put the state vector state, the scheduling behavior action, the reward function reward and the second state vector state_ into the second experience replay pool.
6. The two-stage deep reinforcement learning method for multi-objective dynamic FJSP as claimed in claim 5, characterized in that: Step S6 includes the following sub-steps: In a random manner, extracting data of a preset size from the first experience replay pool replay1, the second experience recycling pool replay2, and the external non-dominated set, and sending the data to the first-order one-layer network and the second-order one-layer network for parameter training; According to the first label value The second label value Performing gradient descent with the network calculation value Q calculated based on the state vector state; The network parameters of the first-order one-layer network and the second-order one-layer network are assigned to the first-order two-layer network and the second-order two-layer network respectively to complete the network update.
