Intelligent workshop real-time rescheduling method based on deep recurrent Q network

Abstract

The application discloses a kind of based on deep cycle Q network's intelligent workshop real-time rescheduling method, it is related to intelligent workshop rescheduling technical field, the based on deep cycle Q network's intelligent workshop real-time rescheduling method includes the following steps: establishing real-time rescheduling sequence decision model;Mapping relationship between real-time state and rescheduling decision of production system is obtained using deep cycle Q network training rescheduling agent;Rescheduling repair action matched with real-time state of production system is executed using trained rescheduling agent;Rescheduling strategy experience data is collected from the replay memory of intelligent workshop and stored in rescheduling agent.The based on deep cycle Q network's intelligent workshop real-time rescheduling method provided by the application can identify the rescheduling point of dynamic production process in real time, and make adaptive rescheduling decision flexibly to offset the negative effects caused by various disturbances.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of intelligent workshop rescheduling, in particular to an intelligent workshop real-time rescheduling method based on a deep recurrent Q network. BACKGROUND

[0002] The diversity of manufacturing resources, the flexibility of product processes and the complexity of production processes pose new challenges to intelligent workshop production operation problems in a dynamic environment. In recent years, under the background of intelligent manufacturing, the development focus of enterprises is shifting from pursuing scale growth to efficiency improvement. Production scheduling and control are important levers for industrial enterprises to continuously optimize production processes and improve workshop production efficiency. In a complex and dynamic production environment, frequent events and disturbances inevitably occur, which can damage the performance and feasibility of the scheduling scheme. How to quickly and economically adjust production tasks according to the real-time changes of production system resources and tasks has become a research hotspot in the field of production operation.

[0003] An intelligent workshop is the central level in the intelligent production process, connecting intelligent factories and intelligent devices. With the application of advanced technologies such as CPS, IoT and digital twinning, the information perception, resource communication and decision execution capabilities of intelligent workshops have been significantly improved, providing a foundation for the continuous improvement of intelligent operation of production processes. In response to this change, some scholars have attempted to combine digital twinning, big data and other technologies with real-time rescheduling problems to study real-time data-driven production control and optimization. In existing research, there are more results about disruptive disturbances, which assume that the occurrence of disturbance events is a sufficient condition for rescheduling. In actual production systems, the cumulative and sudden effects exist simultaneously, and too frequent rescheduling will damage the stability and continuity of the production system. In response to this phenomenon, research usually assumes that workshop information is predictable in a static environment, and uses local information directly related to scheduling performance to develop feature thresholds based on offsets to guide rescheduling decisions, such as completion time, device utilization, etc.

[0004] In an intelligent workshop, based on real-time data integration and interconnection, the production control layer is responsible for the detection and evaluation of the production process, as well as the scheduling and allocation of production resources in the workshop and the decision-making arrangement. The application of frontier technologies provides a better data collection and integration platform for production planning and control, and also puts forward new requirements for solving speed and quality. When the state changes of resources and jobs in the production process are monitored, integrated and forwarded to the control layer in real time, real-time information mining and analysis are needed to evaluate the comprehensive impact of disturbances and seek more adaptive real-time production process control and decision-making mechanisms. However, the impact of disturbances on production systems has multidimensionality and time-varying nature, and predefining a single static feature index threshold cannot meet this requirement. Therefore, a real-time rescheduling integration method based on the strong interactivity and massive real-time data of an intelligent workshop is particularly important. SUMMARY

[0005] In order to solve the problems in the prior art, the application provides an intelligent workshop real-time rescheduling method based on a deep recurrent Q network.

[0006] To this end, the application adopts the following specific technical solutions:

[0007] The intelligent workshop real-time rescheduling method based on the deep recurrent Q network comprises the following steps:

[0008] S1, establishing a real-time rescheduling sequence decision model;

[0009] S2, training a rescheduling agent by using the deep recurrent Q network to obtain a mapping relationship between a production system real-time state and a rescheduling decision;

[0010] S3, executing a rescheduling repair action matched with the production system real-time state by using the trained rescheduling agent;

[0011] S4, collecting rescheduling strategy experience data from a replay memory of the intelligent workshop and storing the rescheduling strategy experience data into the rescheduling agent.

[0012] Preferably, the establishment of the real-time rescheduling sequence decision model comprises the following steps:

[0013] S11, collecting a production system real-time state of the intelligent workshop in real time;

[0014] S12, establishing a real-time rescheduling sequence decision model based on a Markov decision process and using a six-tuple to perform formal expression;

[0015] The expression of the six-tuple is as follows:

[0016] <\mathcal{S,A,T,R,Ω,O}>;

[0017] In the formula, S represents an environment state set; A represents a rescheduling repair action set; O represents an observation of the rescheduling agent; T represents a probability that the production system is transferred from a current environment state s to another state s' after the rescheduling repair action a is executed under the current environment state s; R represents a reward function; and Ω represents a probability that the observation O of the rescheduling agent is obtained after the rescheduling repair action a is executed by the rescheduling agent to make the production system real-time state transferred to another state s';

[0018] S13, taking the rescheduling repair action by the rescheduling agent in the current environment state, making the current environment state transferred to a new environment state based on the transition probability, and obtaining a corresponding reward according to the environment state change.

[0019] Preferably, the training of the rescheduling agent using the deep recurrent Q-network to obtain a mapping relationship between the real-time state of the production system and the rescheduling decision comprises the following steps:

[0020] S21, dividing the production execution process of the smart workshop into continuous time steps, and taking each time step as a rescheduling decision point;

[0021] S22, establishing an offline simulation training environment based on the real-time state of the production system of the smart workshop;

[0022] S23, training the rescheduling agent interactively with the offline simulation training environment to obtain a mapping relationship between the state set of the environment and the rescheduling repair action;

[0023] S24, generating a rescheduling strategy based on the mapping relationship between the state set of the environment and the rescheduling repair action, and storing the rescheduling strategy in the deep recurrent Q-network.

[0024] Preferably, the rescheduling repair action matching the real-time state of the production system is executed by the trained rescheduling agent, comprising the following steps:

[0025] S31, extracting production state features from the real-time state of the production system of the smart workshop;

[0026] S32, executing a rescheduling decision using the rescheduling agent according to the production state features;

[0027] S33, reassigning a specific process of a critical path in the smart workshop to a matching machine processing queue to update the scheduling timetable, completing the real-time rescheduling decision;

[0028] S34, after completing the real-time rescheduling decision, decoding the rescheduling repair action adopted by the rescheduling agent, identifying the corresponding critical path in the smart workshop, and applying the rescheduling repair action to the smart workshop;

[0029] S35, updating the real-time production system state of the smart workshop and executing the corresponding rescheduling repair action.

[0030] Preferably, the production state features include production system stability, MMS offset degree, process proportion, minimum processing time, maximum processing time, maximum left side slack time, minimum left side slack time, maximum right side slack time, and maximum equipment idle time.

[0031] Preferably, the rescheduling strategy experience data includes production system state O t , the rescheduling decision a t selected by the rescheduling agent, the reward r t generated when the production system enters the next time step, and the production system state o t+1Combination (o t a t r t o t+1 ).

[0032] Preferably, the rescheduling repair actions include five rescheduling repair actions, including not performing rescheduling;

[0033] Among them, the first rescheduling repair action is: select the process with the largest LST(i,j,t) in the critical path process set at time t, and assign it to the available equipment set MST of that process. i，j，k，t On the largest device, insert it into the position with the most idle time;

[0034] The second rescheduling and repair action is: selecting the critical path process at time t. or MST i，j，k，t The longest process is assigned to the corresponding equipment and inserted into the position with the longest idle time;

[0035] The third rescheduling repair action is: select the process with the largest LST(i,j,t) in the critical path process set at time t, assign it to the earliest available device in the available device set, and insert it into the earliest available position;

[0036] The fourth rescheduling and repair action is: selecting the critical path process at time t. The largest process is assigned to the equipment with the shortest processing time in the available equipment set and inserted into the position with the longest idle time.

[0037] Preferably, in the rescheduling repair action, LST(i, j, t) represents the process O at time t. i，j The left-side relaxation time is expressed as:

[0038]

[0039] In the formula, LST(i, j, t) represents process O at time t. i，j The left-hand relaxation time; et is an abbreviation for estimate; i and h both represent job indices; j and g represent job J respectively. i and J h Process index; Indicates process O i，j In M k Estimated start time on the machine; Indicates process O h，g Estimated completion time; Y h，g，i，j Indicates process O i，j With process O h，g The relationship.

[0040] Preferably, in the rescheduling repair action, process O i，j With process O h，g The expression for the relationship is:

[0041]

[0042] In the formula, Y h，g，i，j Indicates process O i，j With process O h，g Relationship; MST i，j，k，t Indicates process O at time t i，j The total idle time of available device k within LST(i,j,t), from Start to End; rt k，t Z represents the release / recovery time of available device k at time t; k，t This represents the availability of available device k at time t; Indicates process O h，g The estimated start time.

[0043] Preferably, in the rescheduling repair action, the expression for the availability of available device k at time t is:

[0044]

[0045] In the formula, Z k，t This indicates the availability of available device k at time t.

[0046] The beneficial effects of this invention are as follows:

[0047] 1. The intelligent workshop real-time rescheduling method based on deep cyclic Q-network provided by the present invention can quickly identify rescheduling points in dynamic production processes in real time, flexibly make adaptive rescheduling decisions to offset the negative impacts of various disturbances, and the rescheduling decision scheme can balance scheduling deviation and system stability.

[0048] 2. The real-time rescheduling method for intelligent workshops based on deep cyclic Q-networks provided by this invention has better performance in terms of solution quality and computation time, effectively improving the performance of production operation control. It can coordinate rescheduling of multiple production factors and provide methodological support for the collaborative operation control of intelligent workshops in dynamic environments.

[0049] 3. The real-time rescheduling method for smart workshops based on deep recurrent Q-networks provided by this invention is versatile in production scenarios with different degrees of uncertainty, achieves lean rescheduling cost, has certain economic significance, and thus has the potential to be applied to the actual production and operation of smart workshops. Attached Figure Description

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0051] Figure 1 is a flow chart of an intelligent workshop real-time rescheduling method based on a deep recurrent Q network according to an embodiment of the present application;

[0052] Figure 2 is a specific implementation schematic diagram of an intelligent workshop real-time rescheduling method based on a deep recurrent Q network according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to further illustrate the embodiments, the present application provides drawings which are part of the disclosure of the present application, mainly used to illustrate the embodiments, and can explain the operating principle of the embodiments in conjunction with the related description of the specification. Those skilled in the art should understand other possible implementations and advantages of the present application by referring to these contents.

[0054] According to an embodiment of the present application, an intelligent workshop real-time rescheduling method based on a deep recurrent Q network is provided.

[0055] It should be noted that the intelligent workshop real-time rescheduling method based on a deep recurrent Q network provided by the present application generates a rescheduling scheme for controlling the production process of an intelligent workshop through an agent based on a deep Q network (DQN). The agent of the deep Q network performs the following actions: obtaining the real-time state of the production system of the intelligent workshop, adopting different rescheduling repair actions for the real-time state of the production system, and collecting rescheduling policy experience data during the generation of the scheduling strategy and storing it in the rescheduling agent.

[0056] The present application will be further described in conjunction with the drawings and specific embodiments. As shown in Figure 1 The intelligent workshop real-time rescheduling method based on a deep recurrent Q network according to an embodiment of the present application includes the following steps:

[0057] S1, a real-time rescheduling sequence decision model is established.

[0058] The establishment of the real-time rescheduling sequence decision model includes the following steps:

[0059] S11, the real-time state of the production system of the intelligent workshop is collected in real time;

[0060] S12, a real-time rescheduling sequence decision model is established based on a Markov decision process, and a six-tuple is used for formal expression.

[0061] It should be noted that, considering that complete information of a production system under the influence of multiple disturbances is difficult to obtain in a real production execution process, a Markov decision process with partial observability is used to mathematically express the real-time rescheduling problem.

[0062] The expression of the six-tuple is as follows:

[0063] <mathcal{S,A,T,R,Ω,O}>;

[0064] In the formula, S represents an environment state set; A represents a rescheduling repair action set; O represents an observation of a rescheduling agent; T represents a probability that a production system is transferred from a current environment state s to another state s' after a rescheduling repair action a is performed under the current environment state s, and the expression is as follows:

[0065] T(s, a, s') = P(s'|s, a);

[0066] In the formula, R represents a reward function; and Ω represents a probability that an observation O of a rescheduling agent is obtained after a rescheduling repair action a is performed by the rescheduling agent to make a real-time state of a production system be transferred to another state s'.

[0067] S13, at each time step t, a rescheduling repair action a t ∈A is taken by a rescheduling agent in a current environment state S t ∈S, the current environment state is transferred to a new environment state S t ∈S based on a transition probability T(S t , a t+1 , S t+1 ), and a corresponding reward r t =R(s t , a t ) is obtained according to the change of the environment state, that is, an immediate reward r t is a direct feedback obtained by a real-time rescheduling agent for behavior evaluation, which helps the agent to select a t corresponding to a maximum expected long-term reward.

[0068] It should be noted that S represents the state set of the intelligent workshop production system environment, A represents the action that the agent can select based on the deep recurrent Q network, and O represents the observation of the agent to the production system environment as the basis for the agent to select behavior. The state transition function T(s, a, s') = P(s'|s, a) is used to represent the probability of transferring to state s' after giving action a in environment state s, the observation function Ω(s', a, o) = p(o|s', a) is used to represent the probability of the agent taking action a to make the system state transfer to s' to obtain the observation o, and R represents the reward function R: S x A→ R, that is, the probability of transferring to state s' after giving action a in environment state s.

[0069] S2, training the rescheduling agent using a deep recurrent Q network to obtain the mapping relationship between the real-time state of the production system and the rescheduling decision.

[0070] Wherein, the deep recurrent Q network is used to train the rescheduling agent to obtain the mapping relationship between the real-time state of the production system and the rescheduling decision, comprising the following steps:

[0071] S21, dividing the production execution process of the intelligent workshop into continuous time steps, and taking each time step as a rescheduling decision point;

[0072] S22, establishing an offline simulation training environment based on the real-time state of the production system of the intelligent workshop.

[0073] It should be noted that the offline simulation training environment is established based on the real-time state of the production system of the intelligent workshop, which includes: initializing the intelligent workshop and the production process; initializing the agent based on the deep recurrent Q network; preheating the intelligent workshop and the production process to reach a state suitable for agent learning.

[0074] S23, training the rescheduling agent and the offline simulation training environment to obtain the mapping relationship between the state set S of the environment and the rescheduling repair action a;

[0075] S24, generating a rescheduling strategy π based on the mapping relationship between the state set of the environment and the rescheduling repair action, and storing the rescheduling strategy π in the deep recurrent Q network.

[0076] S3, using the trained rescheduling agent to perform the rescheduling repair action matching the real-time state of the production system.

[0077] Wherein, the trained rescheduling agent is used to perform the rescheduling repair action matching the real-time state of the production system, comprising the following steps:

[0078] S31, extracting production state features from the real-time state of the production system of the intelligent workshop;

[0079] S32, according to the production state characteristics, using the rescheduling agent to execute the rescheduling decision;

[0080] S33, reassigning the specific process of the key path in the smart workshop to the matching machine processing queue to update the scheduling timetable, completing the real-time rescheduling decision;

[0081] S34, after completing the real-time rescheduling decision, decoding the rescheduling repair action adopted by the rescheduling agent, identifying the corresponding key path in the smart workshop, and applying the rescheduling repair action to the smart workshop;

[0082] S35, updating the real-time production system state of the smart workshop and executing the corresponding rescheduling repair action.

[0083] S4, collecting rescheduling strategy experience data from the replay memory of the smart workshop and storing it in the rescheduling agent.

[0084] It should be noted that in order to fully utilize historical training data, improve learning efficiency and training stability, a recurrent network based on LSTM is introduced to perform memory cache replay by using the information of the observation sequence.

[0085] Among them, collecting rescheduling strategy experience data from the replay memory of the smart workshop and storing it in the rescheduling agent includes the following steps:

[0086] In combination with the production system real-time state after the rescheduling repair action is implemented at time t (time t is equivalent to time step t) and the corresponding reward value, the rescheduling strategy experience data (experience scheduling sample) is generated and stored.

[0087] It should be noted that by decoding the rescheduling repair action adopted by the rescheduling agent in step S34, the corresponding key path in the smart workshop is identified, and the rescheduling repair action is applied to the smart workshop, obtaining the next time step state of the smart workshop and the corresponding reward value.

[0088] Among them, the calculation formula of the rescheduling repair action reward value r is:

[0089] r = w c r c +w b r b ;

[0090] In the formula, r c represents the resilience loss index; r b represents the resilience gain index; w c and w b represent the corresponding weights.

[0091] The expression of the resilience loss index is:

[0092]

[0093] wherein, r c denotes the resilience loss index; AR t denotes the action record of the past η time steps at time t, including at; η denotes the number of cumulative time steps calculated from the starting time.

[0094] The expression of the resilience gain index is as follows:

[0095]

[0096] wherein, r b denotes the resilience gain index; denotes the actual maximum completion time at time t; denotes the estimated maximum completion time at time t.

[0097] wherein, the experience scheduling sample is the production system state o t , the rescheduling intelligent agent selects the rescheduling decision a t , the production system enters the next time step to generate the reward r t , and the production system state o t+1 of the next time step. t , a t , r t , o t+1 ).

[0098] A dynamic scheduling model is constructed, and it is determined whether the number of experience scheduling samples exceeds a preset threshold value. If the number of experience scheduling samples exceeds the preset threshold value, a set number of samples are randomly extracted as training samples of the dynamic scheduling model.

[0099] It should be noted that when the number of experience scheduling samples exceeds the preset threshold value Tesample, a set number of samples are randomly extracted as training samples of the dynamic scheduling model. The set number is a small batch to reduce the computational complexity.

[0100] The dynamic scheduling model is trained by using a double-network scheduling model trainer based on Q-learning.

[0101] The dynamic scheduling model is trained by using a double-network scheduling model trainer based on Q-learning, including the following steps:

[0102] An evaluation network is built, and a target network is constructed based on the evaluation network. During the training process of the evaluation network, the parameters of the evaluation network are copied to the target network at a set training interval.

[0103] The evaluation network is built, including the following steps:

[0104] A fully connected deep neural network, consisting of an LSTM input layer, hidden layers, and an output layer, is constructed using a neural network algorithm library. The weights and biases of the fully connected deep neural network are initialized. Training samples are simultaneously input into both the evaluation network and the target network for forward propagation, yielding the behavior value vector Q of the evaluation network. eval With the target network's behavior value vector Q target The evaluation network is backpropagated using the loss function Loss, and its parameters are updated and optimized.

[0105] It should be noted that the loss function Loss is chosen as the Huber loss, and its expression is:

[0106]

[0107] In the formula, L δ (y, f(x)) represents Huber loss; y represents the true value; f(x) represents the predicted value; δ represents the parameters of HuberLoss.

[0108] If the set number of training sessions is reached The trained evaluation network is then output and used as the final dynamic scheduling model. Experience scheduling samples are collected in real time from the playback memory of the smart workshop to update the parameters of the dynamic scheduling model.

[0109] Among them, the rescheduling repair actions include five types of rescheduling repair actions, including not performing rescheduling.

[0110] Among them, the first rescheduling repair action is: select the process with the largest LST(i,j,t) in the critical path process set at time t, and assign it to the available equipment set MST of that process. i，j，k，t On the largest device, insert it into the position with the most idle time.

[0111] The second rescheduling and repair action is: selecting the critical path process at time t. or MST i，j，k，t The longest process is assigned to the corresponding equipment and inserted into the position with the longest idle time.

[0112] The third rescheduling repair action is as follows: Select the process with the largest LST(i,j,t) in the critical path process set at time t, assign it to the earliest available device in the available device set, and insert it into the earliest available position.

[0113] The fourth rescheduling and repair action is: selecting the critical path process at time t. The largest process is assigned to the equipment with the shortest processing time in the available equipment set and inserted into the position with the longest idle time.

[0114] where LST(i, j, t) denotes the left side slack time of operation O i，j at time t, and its expression is:

[0115]

[0116] where LST(i, j, t) denotes the left side slack time of operation O i，j at time t; et is the abbreviation of estimate, which denotes the estimated value; i, h both denote the job index, i, h = 1, 2, …, n; j, g respectively denote the operation index of jobs J i and J h , j = 1, 2, …, n i , g = 1, 2, …, n h ; denotes the estimated start time of operation O i，j on machine M k ; denotes the estimated completion time of operation O h，g ; Y h，g，i，j denotes the relationship of operation O i，j and operation O h，g ; denotes the expected processing time of operation O i，j on machine M k .

[0117] where in the re-scheduling repair action, the expression of the relationship of operation O i，j and operation O h，g is:

[0118]

[0119] where Y h，g，i，j denotes the relationship of operation O i，j and operation O h，g ; MST i，j，k，t denotes the total idle time of available device k for operation O i，j within LST(i, j, t) from the beginning to the end ; rt k，t denotes the release / recovery time of available device k at time t; Z k，t denotes the availability of available device k at time t; denotes the estimated start time of operation O h，g ; if O i.j is a predecessor of O h，g denotes O i，j is a predecessor operation of O h，g ; if Oi.j is a successor of O h，g denotes O i，j is O h，g the successor procedure.

[0120] where, in the re-scheduling repair action, the expression of the availability of the device k at the time step t is:

[0121]

[0122] where, Z k，t denotes the availability of the available device k at time t; if machine k usable at time t denotes that the device k is available at the time step t; if machine k not usable at time t denotes that the device k is not available at the time step t.

[0123] As Figure 2 shown, the following embodiments of the application are further described with the implementation object of the intelligent workshop:

[0124] A coating intelligent workshop model is constructed according to the simplification of the actual production line, which contains 10 devices and can process 8 products, and each workpiece has a fixed process route, and the detailed process data is shown in Table 1.

[0125] Table 1: Detailed process data of workpieces in intelligent workshop

[0126]

[0127]

[0128] The specific implementation process of the real-time rescheduling method of the embodiment is as follows:

[0129] Step one, determine the state space set O and the rescheduling repair action set A of the intelligent workshop, and the specific settings of the two data sets in the embodiment are as follows:

[0130] 1.1: State space set O

[0131] In this embodiment, 9 state features are extracted from the task execution state and the critical path and its slack time to describe the real-time state of the production system, as shown in Table 2.

[0132] Table 2: State space table of intelligent workshop

[0133]

[0134] 1.2 Rescheduling repair action set A

[0135] The embodiment selects five re-scheduling repair actions including not performing re-scheduling, in addition to the not performing re-scheduling action, the other four actions are as described above in the re-scheduling repair action one to the re-scheduling repair action four.

[0136] Step two, build the evaluation deep neural network and the target deep neural network, use Python language and neural network algorithm library such as torch, build a deep neural network containing a 9-dimensional LSTM input layer, 3 hidden layers, an output layer, and a fully connected layer deep neural network containing 32 layer neurons in each hidden layer, and randomly initialize each weight and bias, prepare for forward propagation, wherein the LSTM input layer is set to better process real-time data in the intelligent workshop.

[0137] Step three, simulate the running process of the intelligent workshop, obtain the workshop historical decision information set through simulation, and build an information exchanger on this basis.

[0138] In this embodiment, the process of obtaining the experience scheduling sample on the basis of the information exchanger is summarized as the following steps:

[0139] Initialize the intelligent workshop and the production process and the re-scheduling agent.

[0140] Preheat for 300 time steps to provide normal workshop state for subsequent re-scheduling repair action application.

[0141] Input the real-time workshop state data into the evaluation network for forward propagation, and map the maximum value in the evaluation network output to the re-scheduling repair action set to obtain the current re-scheduling repair action.

[0142] And apply the re-scheduling repair action to the intelligent workshop model, and decode to obtain the production state of the next time step of the workshop and the corresponding reward value.

[0143] The production system state o t , the re-scheduling agent selects the re-scheduling decision a t , the production system enters the next time step to generate the reward r t , and the production system state o t+1 of the next time step are combined into the form of (o t , a t , r t , o t+1 ) set, as an experience scheduling sample, and store the experience scheduling sample in the re-scheduling agent.

[0144] Step four, judge whether the number of experience scheduling samples in the current rescheduling agent reaches the preset threshold Tesample, if the preset threshold is reached, randomly extract a small batch of scheduling samples, and correspondingly output to the evaluation network, the target network and the Q-learning trainer, to prepare data for the training of the evaluation network. If the preset threshold is not reached, repeat steps one to three, the number of small batch scheduling samples in this embodiment is set to 32, and Tesample is set to 1000.

[0145] Step five, the output of the evaluation network is taken as the current action value vector Qeval, and is copied to the intermediate vector Qeval2, the output of the target network is taken as the target action value vector Qtarget, and Qeval2, Qtarget are input into the Q-learning trainer, and the specified elements in the current action value vector Qeval2 are updated through the action value function in the Q-learning algorithm, together with the rescheduling repair action a and the reward value r stored in the trainer. On this basis, a loss function is established, and the evaluation network parameters are updated through back propagation, so as to achieve the purpose of training the evaluation network. At the same time, every certain number of training times, the parameters of the evaluation network are copied to the target network, and in this example, the training number corresponding to the delay copying of the network parameters is set to 100 times.

[0146] Step six, when the training number reaches the preset threshold Tetrain, the evaluation network training is completed, and in this embodiment, Tetrain is set to 100 times. The painting intelligent workshop is connected with the DQN agent, the workshop preheating is 300 time steps, the total processing period is 1000 time steps, and the intelligent workshop running under the guidance of the DQN agent is compared with the training performance of the other three q value based methods, including the final makespan and the total rescheduling number in the production period.

[0147] In summary, with the above technical solutions of the present application, the intelligent workshop real-time rescheduling method based on deep recurrent Q network provided by the present application can quickly identify the rescheduling point of the dynamic production process in real time, flexibly make adaptive rescheduling decisions to offset the negative effects of various disturbances, and the rescheduling decision scheme can balance the scheduling deviation and system stability. The intelligent workshop real-time rescheduling method based on deep recurrent Q network provided by the present application has better performance in solving quality and calculation time, effectively improves the performance of production operation control, and cooperatively reschedules various production elements, providing method support for the cooperative operation control of intelligent workshops in dynamic environment. The intelligent workshop real-time rescheduling method based on deep recurrent Q network provided by the present application has generality in different uncertainty degree production scenes, realizes the lean of rescheduling cost, has certain economic significance, and further has the potential to be applied to the actual production and operation of intelligent workshops.

[0148] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A deep recurrent Q-network-based intelligent workshop real-time rescheduling method, characterized in that, The method comprises: S1, establishing a real-time rescheduling sequence decision model; S2, training a rescheduling agent using a deep recurrent Q network to obtain a mapping relationship between a production system real-time state and a rescheduling decision; S3, using the trained rescheduling agent to perform a rescheduling repair action matched with the production system real-time state; S4, collecting rescheduling strategy experience data from the replay memory of the smart workshop and storing it in the rescheduling agent; The S2 comprises: S21, dividing the production execution process of the smart workshop into continuous time steps, and taking each time step as a rescheduling decision point; S22, establishing an offline simulation training environment based on the real-time state of the production system of the smart workshop; S23, training the rescheduling agent and the offline simulation training environment interactively to obtain a mapping relationship between the state set of the environment and the rescheduling repair action; S24, generating a rescheduling strategy based on the mapping relationship between the state set of the environment and the rescheduling repair action, and storing the rescheduling strategy in the deep recurrent Q network; The S3 comprises: S31, extracting production state features from the real-time state of the production system of the smart workshop; S32, using the rescheduling agent to perform a rescheduling decision according to the production state features; S33, reassigning a specific process of a critical path in the smart workshop to a matched machine processing queue to update the scheduling timetable, and completing real-time rescheduling decision; S34, after completing the real-time rescheduling decision, decoding the rescheduling repair action adopted by the rescheduling agent, identifying the corresponding critical path in the smart workshop, and applying the rescheduling repair action to the smart workshop; S35, updating the real-time production system state of the smart workshop and performing the corresponding rescheduling repair action; The rescheduling repair action comprises no rescheduling, rescheduling repair action one, rescheduling repair action two, rescheduling repair action three, and rescheduling repair action four; The first rescheduling repair action is: select the critical path process set at time t. The largest process is assigned to the centrally available equipment for that process. On the largest device, insert it into the position with the most idle time; The second rescheduling and repair action is: selecting the critical path process at time t. or The longest process is assigned to the corresponding equipment and inserted into the position with the longest idle time; The third rescheduling and repair action is: selecting the critical path process at time t. The largest process is assigned to the earliest available device in the available equipment set and inserted into the earliest available position; The fourth rescheduling repair action is: selecting the key path process in time t from the set of key path processes the largest process, assigning it to the device with the smallest processing time in the set of available devices, and inserting it into the position with the largest idle time; in, Indicates the process at time t The relaxation time on the left side; Indicates the process at time t Available device k in Total idle time within; Indicate process The expected processing time on the machine; i represents the workpiece index; j represents the operation index; et represents the estimate.

2. The intelligent real-time rescheduling method based on deep recurrent Q-network for smart workshop according to claim 1, characterized in that, The establishment of the real-time rescheduling sequence decision model comprises the following steps: S11, real-time collection of the real-time state of the production system of the smart workshop; S12, establishing a real-time rescheduling sequence decision model based on a Markov decision process, and using a six-tuple for formal expression; S13, using the rescheduling agent to take a rescheduling repair action in the current environment state, making the current environment state transition to a new environment state based on transition probability, and obtaining the corresponding reward according to the change of the environment state. 3.The smart workshop real-time rescheduling method based on deep recurrent Q-network according to claim 2, wherein, The production state features comprise production system stability, MMS offset degree, process proportion, minimum processing time, maximum processing time, maximum left side slack time, minimum left side slack time, maximum right side slack time, and maximum equipment idle time.

4. The intelligent real-time rescheduling method based on deep recurrent Q-network for smart workshop according to claim 3, characterized in that, The rescheduling policy experience data includes production system states , rescheduling agent selects rescheduling decision , production system enters next time step produces reward and production system state of next time step combination of , , , .

5. The intelligent real-time rescheduling method based on deep recurrent Q-network for smart workshop according to claim 4, characterized in that, In the re-scheduling repair action, represents the left slack time of the process at time t, which is expressed as: represents the left slack time of the process at time t, which is expressed as: ； wherein, denotes the left side slack time of process at time t; et is an abbreviation of estimate, denoting an estimated value; i, h both denote workpiece indices; j, g respectively denote process indices of workpieces J i and J h ; denotes the estimated start time of process on machine ; denotes the estimated completion time of process ; denotes the relationship of process to process .

6. The intelligent real-time rescheduling method based on deep recurrent Q-network in a smart workshop according to claim 5, characterized in that, In the re-scheduling repair action, the procedure The expression of the relationship between the procedure is: ； wherein represents the process in relation to the process ; represents the process available to the device k in the total free time from the beginning of the time interval to the end of the time interval ; represents the release / recovery time of the device k available at the time t; represents the availability of the device k at the time t; represents the estimated start time of the process ; .

7. The intelligent real-time rescheduling method based on deep recurrent Q-network in a smart workshop according to claim 6, characterized in that, In the rescheduling repair action, the availability of device k at time t is expressed as: ； wherein denotes the availability of device k at time t.

Images