Reinforcement learning environment model construction strategy and scheduling algorithm for optimizing workshop scheduling

Abstract

According to the workshop scheduling optimization method based on reinforcement learning provided by the invention, the optimization effect and generalization ability of a scheduling strategy are remarkably improved by constructing a diversified environment model, dynamically updating model parameters and applying a deep reinforcement learning algorithm. According to the method, the production cost, the production efficiency and the product quality are comprehensively optimized through a multi-target reward function, and real-time adjustment and optimization of a scheduling decision are realized. Specifically, key features are extracted from historical production data of a workshop, and a plurality of environment models reflecting different production situations and scheduling requirements are constructed, so that the generalization ability of a scheduling strategy is enhanced. And then, through deep reinforcement learning methods such as a deep Q-network (DQN), the environment models are pre-trained, so that the intelligent agent can effectively learn and master an optimal scheduling strategy. On the basis, parameters of the environment model are dynamically adjusted through real-time data monitoring and acquisition, so that a scheduling strategy can flexibly adapt to continuously changing production conditions. The method has the advantages that the adaptability of a scheduling strategy to different production situations is remarkably enhanced, the optimization efficiency and the real-time response capability of a scheduling algorithm are improved, the limitation of a traditional method in coping with a complex workshop scheduling problem is successfully overcome, and the method has higher practicability and application value.

Description

Technical Field

[0001] The present invention relates to workshop scheduling technology in the manufacturing industry, and in particular to a scheduling algorithm based on reinforcement learning and a method for constructing an optimization model. Background Art

[0002] The Job Shop Scheduling Problem (JSSP) is a classic combinatorial optimization problem that is widely found in various production workshops in the manufacturing industry. Traditional scheduling methods, such as heuristic algorithms (such as genetic algorithms, ant colony algorithms, etc.) and mathematical programming methods, are often slow to solve or easily fall into the limitations of local optimal solutions, making it difficult to cope with the scheduling needs of complex workshops.

[0003] Reinforcement learning, as a technology for optimizing strategies based on the interaction between intelligent agents and the environment, has received extensive attention in the scheduling field in recent years. In the existing technology, reinforcement learning applications mostly use a single environment model for training, which has problems such as insufficient environment modeling and insufficient generalization of scheduling strategies. Therefore, there is an urgent need for an effective strategy to construct a reinforcement learning environment model to improve the optimization effect and adaptability of the scheduling algorithm. Summary of the invention

[0004] In order to make up for the above shortcomings, the present invention proposes a reinforcement learning environment model construction strategy and scheduling algorithm for optimizing workshop scheduling, including the following steps:

[0005] 1. Data collection and preprocessing: Obtain historical production data from the workshop, including workpiece information (W), machine information (M), processing time (T), process sequence (S), etc. Clean, normalize and extract features from these data to generate a training data set (D) suitable for reinforcement learning.

[0006] Data normalization formula:

[0007]

[0008] Among them, x is the original data and x′ is the normalized data.

[0009] 2. Construction of diversified environment models: According to different production scenarios and scheduling requirements, multiple environment models ({E1, E2, …, E n}). Each model reflects a specific shop floor configuration and workpiece characteristics, such as machine failure probability (pf), workpiece arrival frequency (λ), production priority (P), etc. By generating multiple environments, the generalization ability of the scheduling strategy is increased. The environment model formula is expressed as:

[0010] E i =(M i ,Wi ,T i ,S i ,p f ,λ,P)

[0011] Where E i Represents the i-th environment model.

[0012] 3. Dynamic update of environmental model:

[0013] Real-time monitoring of workshop data (D real-time ), dynamically adjust the parameters of the environmental model according to the latest production status to make it more consistent with the actual production situation, thereby ensuring the effectiveness and accuracy of the scheduling algorithm.

[0014] 4. Scheduling algorithm based on reinforcement learning:

[0015] Reinforcement learning pre-training of agents: Use deep reinforcement learning algorithms (such as deep Q network, DQN) to pre-train the constructed multiple environment models, so that the agent can learn the optimal scheduling strategy from multiple situations. The goal of reinforcement learning is to maximize the cumulative reward in:

[0016]

[0017] Where R t is the cumulative reward at time step t, γ is the discount factor (0<γ<1), r t+k is the immediate reward at time step t+k.

[0018] Trajectory sampling and preference learning: Generate multiple sets of trajectories in different environments, randomly extract segments and calculate their normalized idle time (f idle ) and task completion time (f complete ), applies the preference learning algorithm and generates an optimized scheduling strategy based on the comparison results.

[0019] Calculation of task completion time:

[0020]

[0021] Where n is the total number of tasks, T i is the completion time of the ith task. Optimization of the reward function model: Based on the results of historical data and preference learning, a multi-objective optimization reward function model is established, taking into account production cost (C), production efficiency (E) and product quality (Q). The reward function can be expressed as:

[0022] R=w1·E+w2·Q-w3·C

[0023] Among them, w1, w2, w3 are the weight coefficients of the corresponding targets.

[0024] 5. Real-time scheduling strategy update: Based on new environmental data and optimized reward functions, the scheduling strategy (π) is continuously updated to achieve real-time optimization. The strategy update optimizes the objective function through the gradient ascent method:

[0025]

[0026] Where α is the learning rate.

[0027] 4.2 Scheduling Algorithm Based on Reinforcement Learning

[0028] 1. Reinforcement learning pre-training of intelligent agents: Reinforcement learning pre-training is performed on the constructed multiple environment models. Using methods such as Deep Q-Network (DQN) and Deep Reinforcement Learning (DRL), the intelligent agent can learn the optimal scheduling strategy from multiple scenarios.

[0029] 2. Trajectory sampling and preference learning: Generate multiple sets of trajectories in different environments, randomly extract segments to calculate their normalized idle time and task completion time, apply the preference learning algorithm, and generate an optimized scheduling strategy based on the comparison results.

[0030] 3. Optimization of the reward function model: Based on the results of historical data and preference learning, a multi-objective optimization reward function model is established, which comprehensively considers objectives such as production cost, production efficiency and product quality.

[0031] 4. Real-time scheduling strategy update: Based on new environmental data and optimized reward functions, the scheduling strategy is continuously updated to achieve real-time optimization.

[0032] 4.3 Advantages of the present invention

[0033] ●Improve scheduling optimization effects: Improve the optimization capabilities of scheduling algorithms through diversified environmental model construction and deep training of reinforcement learning.

[0034] ● Enhanced strategy generalization capabilities: Multi-environment training enables scheduling strategies to adapt to different production scenarios, increasing the versatility and robustness of scheduling algorithms.

[0035] ● Strong real-time adaptability: It can dynamically adjust the environmental model and scheduling strategy according to the actual production situation, improving the real-time performance and application value of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 :The flowchart of building a multi-environment model shows how to build multiple environment models according to different production scenarios and scheduling requirements.

[0037] Figure 2:A flowchart of agent training based on reinforcement learning, showing how the agent can perform reinforcement learning training in multiple environments.

[0038] Figure 3 : Scheduling algorithm optimization flowchart describes the overall optimization process of the scheduling algorithm, including data collection, model construction, reinforcement learning training and strategy update. DETAILED DESCRIPTION

[0039] 1. Data Collection and Preprocessing

[0040] 1. Data collection: Obtain historical production data from the workshop management system and production monitoring equipment, including: Workpiece information (such as type, quantity): W = {w1, w2, ..., w n}·Machine information (such as equipment status, processing capacity): M = {m1, m2, ..., m k}·Processing time (such as the average processing time per unit workpiece): T =

[0041] {t ij |i∈W,j∈M}·Process sequence (the priority and process sequence of each task): S=

[0042] {s1,s2,…,s n}2. Data preprocessing: Clean the collected data (remove outliers, handle missing values), normalize and extract features to generate reinforcement learning environment data for training. · Data cleaning: Remove incomplete or erroneous data points and use interpolation or mean to fill missing values. · Data normalization: Use a formula to standardize the data to the range of [0,1] to improve the convergence speed of the algorithm.

[0043]

[0044] 3. Feature extraction: Extract key features (such as processing time, machine load, task sequence, etc.) from the raw data to build the state space (S) and action space (A) required for the reinforcement learning model. State space S =

[0045] W,M,T,S) represents the current production status of the system. The action space A is defined as all possible scheduling decisions.

[0046] 2. Construction of Diversified Environment Model

[0047] 1. Establish multi-environment models: Build multiple simulation environment models E according to different production scenarios and scheduling requirements i ,Each model corresponds to different workshop configurations and production conditions. ·Model parameter setting: such as machine failure probability p f , workpiece arrival frequency λ, production priority P, etc. Mathematical representation of the environment model:

[0048] E i =(M i ,W i ,T i ,S i ,p f ,λ,P)

[0049] 2. Environment Diversification Design: Design multiple environments to reflect different production scenarios, such as high failure rate, low priority tasks, urgent orders, etc. In this way, the generalization ability of the scheduling strategy is increased.

[0050] 3. Dynamic Update of Environmental Model

[0051] 1. Real-time data monitoring and collection: By real-time monitoring of workshop data (such as machine status, workpiece progress, etc.), the parameters of the environmental model are dynamically updated. According to the latest production situation, the model parameters are adjusted, such as updating the machine failure probability p f (t) and the workpiece arrival frequency λ(t). 2. Environmental parameter update strategy: Use sliding window technology to smooth parameter changes and reduce the risk of model overfitting. Sliding window formula:

[0052]

[0053] Where w is the window size. t is the current time step.

[0054] 4. Scheduling Algorithm Based on Reinforcement Learning

[0055] 1. Reinforcement learning pre-training of agents:

[0056] Using reinforcement learning algorithms such as Deep Q Network (DQN), each environment model E i Pre-training is performed to enable the agent to learn how to choose the best scheduling strategy to maximize the cumulative reward R. Objective function:

[0057]

[0058] 2. Trajectory sampling and preference learning:

[0059] Where Q(s,a) is the value function of taking action a in state s, α is the learning rate, and γ is the discount factor. Generate multiple sets of trajectory data in each environment and calculate the idle time f of each set of trajectories idle and task completion time f complete ,The preference learning algorithm is applied to generate the optimized scheduling ,strategy.

[0060] The cumulative reward calculation of the trajectory is:

[0061]

[0062] Where τ represents the trajectory and T is the total number of time steps in the trajectory.

[0063] 3. Optimization of reward function model:

[0064] · Establish a multi-objective optimization reward function model, taking into account production cost C, production efficiency E and product quality Q. · Reward function form:

[0065] R=w1·E+w2·Q-w3·C

[0066] By adjusting the weights w1, w2, and w3, the optimization balance of multiple objectives can be achieved.

[0067] 5. Real-time scheduling strategy update

[0068] 1. Real-time adjustment of scheduling strategy: Based on new environmental data and optimized reward functions, the agent continuously adjusts its scheduling strategy. The strategy update adopts the policy gradient method:

[0069]

[0070] Among them, π is the scheduling strategy and α is the learning rate.

[0071] 2. Real-time monitoring and feedback mechanism: Establish a real-time monitoring system to evaluate the execution effect of the current scheduling strategy and adjust the learning process of the intelligent agent based on feedback to adapt to the dynamic production environment.

Claims

1. A reinforcement learning environment model construction strategy for optimizing workshop scheduling, characterized by include: The steps include environment model construction, agent reinforcement learning pre-training, trajectory sampling and preference learning, reward function model optimization, and real-time scheduling strategy update.

2. The reinforcement learning environment model construction strategy for optimizing workshop scheduling according to claim 1 is characterized in that: The environmental model construction step includes constructing environmental models under multiple different production scenarios.

3. The reinforcement learning environment model construction strategy for optimizing workshop scheduling according to claim 1 is characterized in that: The reward function model is optimized according to multiple objectives such as production cost, production efficiency and product quality.

Images