Job shop scheduling method, apparatus and device, medium and program product
By combining reinforcement learning and ant colony algorithm, the problem that heuristic algorithms in flexible work workshop scheduling is easily trapped in local optimality is solved, more efficient work workshop scheduling is achieved, and better production scheduling plan is provided.
Patent Information
- Application Number
- CN202411164802.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-12
AI Technical Summary
In flexible work workshop scheduling, heuristic intelligent optimization algorithms are prone to fall into local optimal solutions, reducing the efficiency of work workshop scheduling.
Combining reinforcement learning model and ant colony algorithm, reinforcement learning optimizes short-term decisions through interactive learning with the environment, and the ant colony algorithm performs global optimization, using the pheromone system to explore process arrangement and combination, find the optimal production path, and search results of the ant colony algorithm Transformed into reinforcement learning data format for further learning.
Improve the comprehensiveness and efficiency of problem solutions, can respond to complex production scheduling problems more intelligently and flexibly, and obtain better production scheduling plans, providing guidance for the production workshop to continuously generate reasonable and effective operation plans, and improving the operation workshop Scheduling efficiency.
Smart Images

Figure CN118941055A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of manufacturing technology, and in particular to a job shop scheduling method, device, equipment, medium and program product. Background Art
[0002] In modern manufacturing, with the diversification and personalization of customer needs, higher requirements are placed on the flexibility and adaptability of the production process. The flexible job-shop scheduling problem is born in this context. The Flexible Job-shop Scheduling Problem (FJSP) is an extension of the Job-shop Scheduling Problem (JSP). It introduces more flexibility and complexity on the basis of traditional job-shop scheduling. Compared with the traditional job-shop scheduling problem, FJSP allows each job to choose one of multiple machines with the same function for processing, which expands the search range of feasible solutions and provides greater flexibility for production scheduling, but also increases the difficulty of problem solving.
[0003] The core goal of FJSP is to optimize the processing sequence of jobs and the allocation of machines under the premise of satisfying all the processing sequences and resource constraints, so as to achieve optimization goals such as maximizing production efficiency or minimizing production costs. In order to solve the above resource combination optimization problems in production scheduling, a variety of solution algorithms have been developed, mainly including: precise solution methods represented by branch and bound method and mathematical programming method, but such methods are only suitable for solving small-scale problems and are difficult to cope with complex scheduling environments; local optimization methods represented by heuristic rules can quickly establish feasible solutions, but the optimality of the solutions is difficult to guarantee; heuristic intelligent optimization algorithms represented by genetic algorithms, ant colony algorithms, particle swarm algorithms, etc., but each algorithm has a strong dependence on problem characteristics, and the search operation requires special design, which is easy to fall into local optimality, and it is difficult to obtain satisfactory solutions within a limited time when facing large-scale scheduling problems. Summary of the invention
[0004] The embodiments of the present application provide a job shop scheduling method, device, equipment, medium and program product to solve the technical problem in the field of flexible job shop scheduling in the related technology that the production scheduling plan obtained by the heuristic intelligent optimization algorithm is prone to fall into the local optimal solution, thereby reducing the efficiency of job shop scheduling.
[0005] In a first aspect, an embodiment of the present application provides a job shop scheduling method, the method comprising:
[0006] Acquire the operation information of the target workshop, wherein the operation information includes: information of each process of each pending order of the target workshop, and information of optional equipment for each process;
[0007] The job information is input into the reinforcement learning model, and an iteration step is performed, wherein the iteration step includes: iteratively running the reinforcement learning model, performing multiple actions in each iteration of the reinforcement learning model, obtaining multiple environmental states and corresponding immediate rewards, and each execution of an action generates a corresponding state sequence, and the state sequence is added to a preset experience pool, wherein the environmental state is: the process progress of the order to be processed and the equipment information selected for the process, and the state sequence includes: the environmental state before executing the action, the environmental state after executing the action, the immediate reward corresponding to the execution of the action, and the action corresponding to the execution of the action; the action is: determining the equipment information to be selected for the current process;
[0008] After the current reinforcement learning model is iterated for a first preset number of times, the current ant colony algorithm model is run once, the input of the ant colony algorithm model is the job information, and the output is a plurality of production scheduling sequences of all processes in all pending orders; and the optimal production scheduling sequence is determined from the plurality of production scheduling sequences;
[0009] After converting the optimal production scheduling sequence into the target state sequence of the reinforcement learning model, adding the target state sequence to the experience pool;
[0010] Model updating step: determining whether the number of state sequences in the experience pool reaches a second preset number, and if so, updating the parameters of the reinforcement learning model based on the state sequences in the experience pool to obtain an updated reinforcement learning model; and updating the pheromone of the ant colony algorithm model after each operation of the ant colony algorithm model to obtain an updated ant colony algorithm model;
[0011] Repeating the iterative step and the model updating step until the total number of iterations reaches a third preset number;
[0012] Determine an optimal environment state from all state sequences of the reinforcement learning model, wherein the maximum completion time of all processes of all pending orders corresponding to the optimal environment state is the smallest;
[0013] The job shop is scheduled based on the optimal environmental state.
[0014] Optionally, the immediate reward corresponding to the execution action is determined by the pheromone concentrations between the path nodes corresponding to the environment state before the execution action and the environment state after the execution action in the ant colony algorithm model.
[0015] Optionally, the instant reward corresponding to the execution action is determined by the pheromone concentrations between the path nodes corresponding to the environment state before the execution action and the environment state after the execution action in the ant colony algorithm model, and is implemented by the following formula:
[0016] r t =-ω i,k +τ xy ; C t+1 >C t ;
[0017] r t =ω i,k +τ xy ; C t+1 ≤C t ;
[0018] Among them, the r t is the immediate reward corresponding to the execution action; xy is the pheromone concentration;
[0019] The C t is the maximum completion time corresponding to the environmental state before the execution of the action, and the C t+1 is the maximum completion time corresponding to the environmental state after the execution of the action; i,k is the kth process O of order i i,k The processing time is , where i and k are both positive integers.
[0020] Optionally, the method further includes:
[0021] After each iteration of the current reinforcement learning model is run once, a global reward factor corresponding to this iteration is determined;
[0022] According to the global reward factor corresponding to the current iteration, the immediate reward corresponding to the execution action in the state sequence in the experience pool is updated to obtain an updated immediate reward, wherein the updated immediate reward is used to influence the next iteration of the reinforcement learning model.
[0023] Optionally, according to the global reward factor corresponding to the current iteration, the immediate reward corresponding to the execution action in the state sequence in the experience pool is updated, and the updated immediate reward is obtained by the following formula:
[0024]
[0025] Among them, the r t is the immediate reward corresponding to the execution action, and the r t+1 is the updated instant reward, is the global reward factor corresponding to this iteration, C_max is the maximum completion time of all processes of all pending orders corresponding to this iteration; Q is the preset reward function corresponding to the reinforcement learning model.
[0026] In a second aspect, an embodiment of the present application provides a job shop scheduling device, the device comprising:
[0027] An acquisition module, used for acquiring operation information of a target workshop, wherein the operation information includes: information of each process of each pending order of the target workshop, and information of optional equipment for each process;
[0028] An execution module is used to input the job information into the reinforcement learning model and execute an iteration step, wherein the iteration step includes: iteratively running the reinforcement learning model, executing multiple actions in each iteration of the reinforcement learning model, obtaining multiple environmental states and corresponding immediate rewards, and generating a corresponding state sequence each time an action is executed, and adding the state sequence to a preset experience pool, wherein the environmental state is: the process progress of the order to be processed and the equipment information selected for the process, and the state sequence includes: the environmental state before executing the action, the environmental state after executing the action, the immediate reward corresponding to the execution of the action, and the action corresponding to the execution of the action; the action is: determining the equipment information to be selected for the current process;
[0029] After the current reinforcement learning model is iterated for a first preset number of times, the current ant colony algorithm model is run once, the input of the ant colony algorithm model is the job information, and the output is a plurality of production scheduling sequences of all processes in all pending orders; and the optimal production scheduling sequence is determined from the plurality of production scheduling sequences;
[0030] After converting the optimal production scheduling sequence into the target state sequence of the reinforcement learning model, adding the target state sequence to the experience pool;
[0031] Model updating step: determining whether the number of state sequences in the experience pool reaches a second preset number, and if so, updating the parameters of the reinforcement learning model based on the state sequences in the experience pool to obtain an updated reinforcement learning model; and updating the pheromone of the ant colony algorithm model after each operation of the ant colony algorithm model to obtain an updated ant colony algorithm model;
[0032] Repeating the iterative step and the model updating step until the total number of iterations reaches a third preset number;
[0033] Determine an optimal environment state from all state sequences of the reinforcement learning model, wherein the maximum completion time of all processes of all pending orders corresponding to the optimal environment state is the smallest;
[0034] The job shop is scheduled based on the optimal environmental state.
[0035] Optionally, the immediate reward corresponding to the execution action is determined by the pheromone concentrations between the path nodes corresponding to the environment state before the execution action and the environment state after the execution action in the ant colony algorithm model.
[0036] Optionally, the instant reward corresponding to the execution action is determined by the pheromone concentrations between the path nodes corresponding to the environment state before the execution action and the environment state after the execution action in the ant colony algorithm model, and is implemented by the following formula:
[0037] r t =-ω i,k +τ xy ; C t+1 >C t ;
[0038] r t =ω i,k +τ xy ; C t+1 ≤C t ;
[0039] Among them, the r t is the immediate reward corresponding to the execution action; xy is the pheromone concentration;
[0040] The C t is the maximum completion time corresponding to the environmental state before the execution of the action, and the C t+1 is the maximum completion time corresponding to the environmental state after the execution of the action; i,k is the kth process O of order i i,k The processing time is , where i and k are both positive integers.
[0041] Optionally, the execution module is also used to determine the global reward factor corresponding to the current iteration after each iteration of the current reinforcement learning model is run once; according to the global reward factor corresponding to the current iteration, the instant reward corresponding to the execution action in the state sequence in the experience pool is updated to obtain an updated instant reward, wherein the updated instant reward is used to influence the next iteration of the reinforcement learning model.
[0042] Optionally, according to the global reward factor corresponding to the current iteration, the immediate reward corresponding to the execution action in the state sequence in the experience pool is updated, and the updated immediate reward is obtained by the following formula:
[0043]
[0044] Among them, the r t is the immediate reward corresponding to the execution action, and the r t+1 is the updated instant reward, is the global reward factor corresponding to this iteration, C_max is the maximum completion time of all processes of all pending orders corresponding to this iteration; Q is the preset reward function corresponding to the reinforcement learning model.
[0045] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of a job shop scheduling method as described in the first aspect are implemented.
[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of a job shop scheduling method as described in the first aspect are implemented.
[0047] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of a job shop scheduling method as described in the first aspect.
[0048] Therefore, the embodiment of the present application proposes a method for scheduling a job shop that combines a reinforcement learning model and an ant colony algorithm. Reinforcement learning is responsible for learning from each interaction with the environment, optimizing short-term decisions, and updating the decision model by evaluating the impact of different actions (such as assigning a specific process to a certain device) on the goal (such as minimizing the maximum completion time). Reinforcement learning continuously learns through trial and error, and gradually improves the quality of decision-making. The ant colony algorithm is used for global optimization. By simulating the process of ants looking for food, all possible process permutations and combinations are explored to find the optimal production path. The ant colony algorithm uses a pheromone system to record historically effective paths to help later ants make better path choices. After a certain number of iterations of reinforcement learning, the ant colony algorithm is executed once, and the data accumulated by reinforcement learning so far is used for global search. The search results of the ant colony algorithm (i.e., the optimal path) are converted into a data format available for reinforcement learning and input back into the experience replay pool, so that reinforcement learning can continue to learn from these global optimal solutions. As a result, not only can optimization be performed in local decisions (through reinforcement learning), but also the best operating strategy can be found at the global level (through the ant colony algorithm). This integrated approach improves the comprehensiveness and efficiency of problem-solving solutions, enabling the scheduling system to respond to complex production scheduling problems more intelligently and flexibly, thereby deriving better production scheduling plans, providing guidance for production workshops to continuously generate reasonable and effective job plans and schedules, and improving the efficiency of job workshop scheduling. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0050] Figure 1 A flowchart of a method for job shop scheduling provided in an embodiment of the present application;
[0051] Figure 2 A flowchart of a method for job shop scheduling provided in an embodiment of the present application;
[0052] Figure 3 The architecture diagram of the reinforcement learning model and the ant colony algorithm model provided in the embodiments of the present application;
[0053] Figure 4 A structural block diagram of a device for job shop scheduling provided in an embodiment of the present application;
[0054] Figure 5 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0056] Figure 1 A job shop scheduling method provided according to an embodiment of the present application is shown, the method comprising:
[0057] Step S101, obtaining operation information of a target workshop;
[0058] The operation information includes: information about each process of each pending order in the target workshop, and information about the optional equipment for each process;
[0059] Step S102: inputting the operation information into the reinforcement learning model and executing the iterative step;
[0060] Among them, the iteration step includes: iteratively running the reinforcement learning model, in each iterative running of the reinforcement learning model, executing multiple actions, obtaining multiple environmental states and corresponding immediate rewards, each execution of an action will generate a corresponding state sequence, and the state sequence is added to a preset experience pool; wherein, the environmental state is: the process progress of the pending order and the equipment information selected for the process, and the state sequence includes: the environmental state before executing the action, the environmental state after executing the action, the immediate reward corresponding to the action, and the action corresponding to the action; the action is: determining the equipment information that should be selected for the current process; after the current reinforcement learning model is iteratively run for a first preset number of times, the current ant colony algorithm model is run once, the input of the ant colony algorithm model is the job information, and the output is a variety of production scheduling sequences for all processes in all pending orders; and from the multiple production scheduling sequences, the optimal production scheduling sequence is determined; after the optimal production scheduling sequence is converted into the target state sequence of the reinforcement learning model, the target state sequence is added to the experience pool;
[0061] Step S103, executing a model updating step, the model updating step comprising: determining whether the number of state sequences in the experience pool reaches a second preset number, and if so, updating the parameters of the reinforcement learning model based on the state sequences in the experience pool to obtain an updated reinforcement learning model; and after each operation of the ant colony algorithm model, updating the pheromone of the ant colony algorithm model to obtain an updated ant colony algorithm model;
[0062] Step S104, repeatedly executing the iteration step and the model updating step until the total number of iterations reaches a third preset number;
[0063] Step S105: Determine the optimal environment state from all state sequences of the reinforcement learning model;
[0064] Among them, the maximum completion time of all processes of all pending orders corresponding to the optimal environmental state is the smallest;
[0065] Step S106: Scheduling the job shop based on the optimal environmental state.
[0066] In a possible implementation, the immediate reward corresponding to the execution of the action is determined by the pheromone concentrations between the path nodes corresponding to the environment state before the execution of the action and the environment state after the execution of the action in the ant colony algorithm model.
[0067] In a possible implementation, the immediate reward corresponding to the action is determined by the pheromone concentration between the corresponding path nodes in the ant colony algorithm model according to the environment state before and after the action is performed, and is implemented by the following formula:
[0068] r t =-ω i,k +τ xy ; C t+1 >C t ;
[0069] r t =ω i,k +τ xy ; C t+1 ≤C t ;
[0070] Among them, r t is the immediate reward corresponding to the action; τ xy is the pheromone concentration;
[0071] C t is the maximum completion time corresponding to the environment state before executing the action, C t+1 is the maximum completion time corresponding to the environmental state after the action is executed; ω i,k is the kth process O of order i i,k The processing time is , where i and k are both positive integers.
[0072] In a possible implementation, the method further includes: after each iteration of the current reinforcement learning model is run once, determining a global reward factor corresponding to this iteration; updating the immediate reward corresponding to the execution action in the state sequence in the experience pool according to the global reward factor corresponding to this iteration to obtain an updated immediate reward, wherein the updated immediate reward is used to influence the next iteration of the reinforcement learning model.
[0073] In one possible implementation, the immediate reward corresponding to the execution action in the state sequence in the experience pool is updated according to the global reward factor corresponding to this iteration, and the updated immediate reward is realized by the following formula:
[0074]
[0075] Among them, r t is the immediate reward corresponding to the action, r t+1 For the updated instant rewards, is the global reward factor corresponding to this iteration, C_max is the maximum completion time of all processes of all pending orders corresponding to this iteration; Q is the preset reward function corresponding to the reinforcement learning model.
[0076] Therefore, the embodiment of the present application proposes a method for scheduling a job shop that combines a reinforcement learning model and an ant colony algorithm. Reinforcement learning is responsible for learning from each interaction with the environment, optimizing short-term decisions, and updating the decision model by evaluating the impact of different actions (such as assigning a specific process to a certain device) on the goal (such as minimizing the maximum completion time). Reinforcement learning continuously learns through trial and error, and gradually improves the quality of decision-making. The ant colony algorithm is used for global optimization. By simulating the process of ants looking for food, all possible process permutations and combinations are explored to find the optimal production path. The ant colony algorithm uses a pheromone system to record historically effective paths to help later ants make better path choices. After a certain number of iterations of reinforcement learning, the ant colony algorithm is executed once, and the data accumulated by reinforcement learning so far is used for global search. The search results of the ant colony algorithm (i.e., the optimal path) are converted into a data format available for reinforcement learning and input back into the experience replay pool, so that reinforcement learning can continue to learn from these global optimal solutions. As a result, not only can optimization be performed in local decisions (through reinforcement learning), but also the best operating strategy can be found at the global level (through the ant colony algorithm). This integrated approach improves the comprehensiveness and efficiency of problem-solving solutions, enabling the scheduling system to respond to complex production scheduling problems more intelligently and flexibly, thereby deriving better production scheduling plans, providing guidance for production workshops to continuously generate reasonable and effective job plans and schedules, and improving the efficiency of job workshop scheduling.
[0077] Now, from the perspective of building a reinforcement learning model and an ant colony algorithm model, the method provided by the embodiment of the present application is introduced in detail. The method can be referred to Figure 2 , methods include:
[0078] Step 1: Initialize the scheduling model and build a parallel framework of reinforcement learning and ant colony algorithm (such as Figure 3 shown).
[0079] Specifically include:
[0080] Step 1a: Construct a workshop environment state model. The workshop environment includes order task scheduling progress information and workshop equipment occupancy information, which is expressed as S = (S o ,S m ).in Indicates the order collection has been scheduled progress status information, Indicates the current process schedule of order i (i = 1, 2, 3...N, N is the number of orders), and the value range is [0, K i ], K i is the total number of processes for order i; Indicates the time status information of the equipment collection. Indicates the status of equipment j (j = 1, 2, 3...M, M is the number of equipment) currently occupied by scheduled process tasks, such as Figure 2 As shown in 1a-c. For example: the state of equipment being occupied by scheduled process tasks means that if a piece of equipment is occupied by process 1 of order 1, and the working time of this process is 5 hours, then the state of the equipment being occupied by the scheduled process tasks is 5 hours. If process 2 of order 3 (working time is 3 hours) is scheduled on the equipment, then the state of the equipment being occupied by the scheduled process tasks is 5+3=8 hours.
[0081] Step 1b: For the two decision-making processes of process selection and equipment arrangement in flexible workshop scheduling, two independent DQN (Deep Q-Network) learning processes are established in the agent. o and Q m They are responsible for two decision-making processes respectively, thus constructing a two-layer Markov decision process model. Among them, the process selection is represented by a 4-tuple Representation; equipment arrangement uses 4-tuple The specific meanings are as follows: and Represents the state at time step t. The agent interacts with the environment according to a specific strategy π(S→A) and performs actions and Get the new status and And get reward t (r t ∈R t ).
[0082] Step 1c: Initialize the workshop environment state S1; initialize various parameter settings, including the number of sequence sampling times E, the number of network layers, and the random parameter initialization network Q o , Q m ; Copy the same parameters to the target network Create two experience replay pools buffer1 and buffer2 for the two networks; the number of sampled data is N batch ; Greedy strategy value ∈; Set the production scheduling objective function. It should be noted that the embodiment of the present application adopts minimizing the maximum completion time min (C-max) of the order as the production scheduling objective.
[0083] Step 1d: Initialize the ant colony algorithm and establish the ant colony algorithm structure for the process selection process in flexible workshop scheduling, which is responsible for optimizing the process sorting order, such as Figure 2 As shown in 1d. The total number of iterations is equal to the number of sampling times E, the path node is the process set P of all orders, and the process is represented by O m,n (represents the nth process of order m, where m and n are both positive integers), the number of ants is s, the volatility coefficient is ρ, the pheromone intensity is Q, and each ant starts from the initial node O start Start from the node, and complete the path finding after passing through all process nodes. The order of the paths passed is the order of process arrangement.
[0084] For the equipment selection process, heuristic rules are used to arrange the process, compare the estimated completion time of each process on all optional equipment, and select the earliest completed equipment for arrangement. Initialize the pheromone record table, the row label X in the table is the starting node, the column label Y in the table is the target node, and the value τ in the table xy Used to record the cumulative pheromone concentration values left by all ants from x→y.
[0085] Among them, heuristic rules refer to a method based on specific logic or algorithm to simplify the decision-making process, which is used to quickly find effective solutions in complex decision-making environments. In flexible shop scheduling, especially for equipment selection, heuristic rules are often used to evaluate and select the optimal equipment to perform a specific process.
[0086] Step 2: Perform algorithm iteration. The reinforcement learning process obtains (s t ,a t ,r t ,s t+1 ) sequence and save it to the experience replay pool. After T steps of reinforcement learning iteration (which can be customized by the user according to the actual operation of the model), the ant colony algorithm will store the generated s results in a fixed-length solution set after one iteration, and select the optimal solution, which is parsed as (s t ,a t ,r t ,s t+1 ) sequence and update it synchronously to the experience replay pool.
[0087] Specifically include:
[0088] Step 2a: Figure 2 As shown in 2a, the reinforcement learning process is in the current state S t Next, according to the current network Q o and Q m , choose actions with ∈-greedy strategy and The method for selecting an action is as follows:
[0089]
[0090] Step 2b: The workshop environment model takes actions based on the agent and updates the state to and And return the reward r t (r t ∈R t ),like Figure 2 As shown in 2b.
[0091] The order status is updated as follows: The order status is but
[0092] The device status is updated as follows: The current status of the device is but
[0093] Reward t Determined according to the state changes before and after, if S t The maximum completion time of all scheduled tasks on the device is C t , execute the above two actions, suppose the kth process O of order i is selected i,k The processing time is ω i,k , arrange it on device j, then S t+1 The maximum completion time of all scheduled tasks on the device is C t+1 , and find the state S in the pheromone table t →S t+1 Corresponding process node transfer path pheromone concentration τ xy , judgement: if C t+1 >C t , r t =-ω i,k +τ xy ; otherwise r t =ω i,k +τ xyTherefore, the pheromone concentration is used to adjust the immediate reward value, and the two-layer reinforcement learning and ant colony algorithm work together to combine their respective advantages to achieve the technical effect of ultimately optimizing the overall scheduling efficiency of the production workshop. After that, the sequence data generated by sampling is stored in two corresponding experience replay pools buffer1 and buffer2 respectively.
[0094] Step 2c: Execute the sampling process in 2a above until the sampling of the production scheduling process sequence is completed and a production scheduling result is obtained. After analysis, the production scheduling target value is C_max. Then set is the global reward factor of the sequence, which is added to the corresponding sequence reward in the experience replay pool, such as Figure 2 As shown in 2c, the update formula is:
[0095]
[0096] Step 2d: Figure 2 As shown in Figure 2d, after T steps of reinforcement learning iteration, the ant colony algorithm is executed once, the number of ants is s, and each ant starts from the initial node O start After all the process nodes are passed, a solution is completed. When the ant transfers from node x to the next node y, only the path pheromone concentration τ is considered to simplify the calculation xy , then the transition probability is:
[0097]
[0098] Where allowed is the set of all unscheduled process nodes that the ants can reach from node x. After all ants have completed a path optimization, all solutions are added to the solution set D, and then all solutions in D are sorted by target value to select the optimal solution d best , and then decode it and parse it into a double layer (s t ,a t ,r t ,s t+1 ) sequence, as in step 2a, and saved to the experience replay pools buffer1 and buffer2.
[0099] Step 3: Update network parameters and pheromone table.
[0100] Step 3a: For the reinforcement learning model, when there is enough data in the replay pool buffer1 and buffer2 (that is, when the data is greater than the required number of samples, sampling can be performed), the number of samples from each is N batch Sequence data like Figure 2 As shown in 3a, the target network is then used to calculate each data Then minimize the loss function L constructed in the form of mean square error, and use the gradient descent method to update the current network Q. After each iteration T steps, the target network synchronizes parameters with the current training network.
[0101]
[0102] Step 3b: Update the pheromone table of the ant colony algorithm. Update all solution paths in the solution set D according to the pheromone update formula of the traditional ant colony algorithm (the following two formulas), as follows: Figure 2 As shown in Figure 3b.
[0103] τ xy (t+1)=(1-ρ)τ xy (t)+∑ s Δτ xy (t,t+1);
[0104]
[0105] Step 4: Determine whether the termination condition is met. If the termination condition is met, that is, Ne ≥ E, the loop ends, the reinforcement learning model parameters are saved, and the historical optimal solution is output; if not, Ne = Ne + 1, go to step 2.
[0106] It should be noted that with the development of machine learning technologies such as reinforcement learning and deep learning, combining them with traditional heuristic algorithms can improve the intelligence level of scheduling and the generalization ability of scheduling strategies, so that they can adapt to different production environments and scales. Reinforcement learning can learn the best behavior strategy through interaction with the environment in sequential decision-making problems, which is consistent with the task sequence arrangement decision-making characteristics of workshop scheduling problems. Reinforcement learning has a global optimization perspective, can automatically adjust strategies according to real-time feedback, adapt to environmental changes, has better robustness, has real-time decision-making capabilities in response efficiency, can quickly make scheduling decisions in real-time environments, and has good generalization and expansion capabilities.
[0107] In summary, the embodiment of the present application proposes a job shop scheduling (flexible job shop scheduling) method that combines reinforcement learning (two-layer reinforcement learning) and ant colony algorithm. The method proposes to transform the process sorting and equipment selection problems in the flexible job shop scheduling process into a two-layer Markov decision process to realize the model construction of reinforcement learning; it proposes to use the reinforcement learning algorithm to realize the interaction between the intelligent agent and the workshop environment, and complete the process arrangement based on the current workshop state with the goal of maximizing the cumulative reward; it proposes a cumulative reward feedback method that imitates the pheromone update mechanism of the ant colony algorithm, and uses the accumulated pheromones on the production scheduling path as an additional reward for the reinforcement learning reward function to optimize the algorithm convergence speed; it proposes a parallel scheme for the ant colony algorithm and the reinforcement learning process, and analyzes the optimal solution obtained during the operation of the ant colony algorithm into an expert data sample sequence, which is added to the reinforcement learning process to realize the diversification of the exploration process.
[0108] Figure 4 A job shop scheduling device 40 provided in an embodiment of the present application is shown, and the device 40 includes:
[0109] The acquisition module 401 is used to acquire the operation information of the target workshop, wherein the operation information includes: information of each process of each pending order of the target workshop, and information of optional equipment for each process;
[0110] The execution module 402 is used to input the operation information into the reinforcement learning model and execute the iteration step, which includes: iteratively running the reinforcement learning model, executing multiple actions in each iteration of the reinforcement learning model, obtaining multiple environmental states and corresponding immediate rewards, and generating a corresponding state sequence each time an action is executed, and adding the state sequence to a preset experience pool, wherein the environmental state is: the process progress of the order to be processed and the equipment information selected for the process, and the state sequence includes: the environmental state before executing the action, the environmental state after executing the action, the immediate reward corresponding to the execution of the action, and the action corresponding to the execution of the action; the action is: determining the equipment information to be selected for the current process;
[0111] After the current reinforcement learning model is iterated for a first preset number of times, the current ant colony algorithm model is run once, the input of the ant colony algorithm model is the job information, and the output is a plurality of production scheduling sequences of all processes in all pending orders; and the optimal production scheduling sequence is determined from the plurality of production scheduling sequences;
[0112] After converting the optimal production scheduling sequence into the target state sequence of the reinforcement learning model, the target state sequence is added to the experience pool;
[0113] Model updating step: determining whether the number of state sequences in the experience pool reaches a second preset number, and if so, updating the parameters of the reinforcement learning model based on the state sequences in the experience pool to obtain an updated reinforcement learning model; and after each operation of the ant colony algorithm model, updating the pheromone of the ant colony algorithm model to obtain an updated ant colony algorithm model;
[0114] Repeating the iteration step and the model updating step until the total number of iterations reaches a third preset number;
[0115] Determine the optimal environment state from all state sequences of the reinforcement learning model, where the maximum completion time of all processes of all pending orders corresponding to the optimal environment state is the smallest;
[0116] Scheduling of job shops based on optimal environmental conditions.
[0117] In a possible implementation, the immediate reward corresponding to the execution of the action is determined by the pheromone concentrations between the path nodes corresponding to the environment state before the execution of the action and the environment state after the execution of the action in the ant colony algorithm model.
[0118] In a possible implementation, the immediate reward corresponding to the action is determined by the pheromone concentration between the corresponding path nodes in the ant colony algorithm model according to the environment state before and after the action is performed, and is implemented by the following formula:
[0119] r t =-ω i,k +τ xy ; C t+1 >C t ;
[0120] r t =ω i,k +τ xy ; C t+1 ≤C t ;
[0121] Among them, r t is the immediate reward corresponding to the action; τ xy is the pheromone concentration;
[0122] C t is the maximum completion time corresponding to the environment state before executing the action, C t+1 is the maximum completion time corresponding to the environmental state after the action is executed; ω i,k is the kth process O of order i i,k The processing time is , where i and k are both positive integers.
[0123] In a possible implementation, the execution module 402 is further used to determine the global reward factor corresponding to the current iteration after each iteration of the current reinforcement learning model is run once; according to the global reward factor corresponding to the current iteration, the immediate reward corresponding to the execution action in the state sequence in the experience pool is updated to obtain an updated immediate reward, wherein the updated immediate reward is used to influence the next iteration of the reinforcement learning model.
[0124] In one possible implementation, the immediate reward corresponding to the execution action in the state sequence in the experience pool is updated according to the global reward factor corresponding to this iteration, and the updated immediate reward is realized by the following formula:
[0125]
[0126] Among them, r t is the immediate reward corresponding to the action, r t+1 For the updated instant rewards, is the global reward factor corresponding to this iteration, C_max is the maximum completion time of all processes of all pending orders corresponding to this iteration; Q is the preset reward function corresponding to the reinforcement learning model.
[0127] In summary, the embodiments of the present application propose a parallel algorithm structure that combines two-layer reinforcement learning and an ant colony algorithm, which can effectively combine the local optimization capability of the ant colony algorithm and the global adaptability of reinforcement learning, and update the adjustment strategy in real time; propose a method for converting the flexible job shop scheduling process into a two-layer Markov decision process, and the environmental state can simultaneously reflect the task processing progress and equipment load changes, and the intelligent agent determines the task arrangement through a two-layer decision process; propose a design method for a reinforcement learning reward function that considers scheduling objectives and a pheromone additional reward mechanism, and feed back the global scheduling objective information into the reward function to improve the algorithm's global optimization ability and convergence speed.
[0128] The present application embodiment also provides an electronic device 50, such as Figure 5 As shown, it includes: a processor 501, a memory 502, and a program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor, the steps of the job shop scheduling method shown in the above embodiment are implemented.
[0129] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the job shop scheduling method shown in the above method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0130] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the job shop scheduling method shown in the above method embodiment and can achieve the same technical effect. To avoid repetition, they are not repeated here.
[0131] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0133] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A job shop scheduling method, characterized in that: The method comprises: Acquire the operation information of the target workshop, wherein the operation information includes: information of each process of each pending order of the target workshop, and information of optional equipment for each process; The job information is input into the reinforcement learning model, and an iteration step is performed, wherein the iteration step includes: iteratively running the reinforcement learning model, performing multiple actions in each iteration of the reinforcement learning model, obtaining multiple environmental states and corresponding immediate rewards, and each execution of an action generates a corresponding state sequence, and the state sequence is added to a preset experience pool, wherein the environmental state is: the process progress of the order to be processed and the equipment information selected for the process, and the state sequence includes: the environmental state before executing the action, the environmental state after executing the action, the immediate reward corresponding to the execution of the action, and the action corresponding to the execution of the action; the action is: determining the equipment information to be selected for the current process; After the current reinforcement learning model is iterated for a first preset number of times, the current ant colony algorithm model is run once, the input of the ant colony algorithm model is the job information, and the output is a plurality of production scheduling sequences of all processes in all pending orders; and the optimal production scheduling sequence is determined from the plurality of production scheduling sequences; After converting the optimal production scheduling sequence into the target state sequence of the reinforcement learning model, adding the target state sequence to the experience pool; Model updating step: determining whether the number of state sequences in the experience pool reaches a second preset number, and if so, updating the parameters of the reinforcement learning model based on the state sequences in the experience pool to obtain an updated reinforcement learning model; and updating the pheromone of the ant colony algorithm model after each operation of the ant colony algorithm model to obtain an updated ant colony algorithm model; Repeating the iterative step and the model updating step until the total number of iterations reaches a third preset number; Determine an optimal environment state from all state sequences of the reinforcement learning model, wherein the maximum completion time of all processes of all pending orders corresponding to the optimal environment state is the smallest; The job shop is scheduled based on the optimal environmental state.
2. The method according to claim 1, characterized in that The immediate reward corresponding to the execution action is determined by the pheromone concentrations between the path nodes corresponding to the environment state before the execution action and the environment state after the execution action in the ant colony algorithm model.
3. The method according to claim 2, characterized in that The instant reward corresponding to the execution action is determined by the pheromone concentrations between the corresponding path nodes in the ant colony algorithm model according to the environmental state before the execution action and the environmental state after the execution action, which is implemented by the following formula: r t =-ω i,k +t xy ;C t+1 >C t ; r t =ω i,k +t xy ;C t+1 ≤C t ; Among them, the r t is the immediate reward corresponding to the execution action; xy is the pheromone concentration; The C t is the maximum completion time corresponding to the environmental state before the execution of the action, and the C t+1 is the maximum completion time corresponding to the environmental state after the execution of the action; i,k is the kth process O of order i i,k The processing time is , where i and k are both positive integers.
4. The method according to claim 1 or 2, characterized in that: The method further comprises: After each iteration of the current reinforcement learning model is run once, a global reward factor corresponding to this iteration is determined; According to the global reward factor corresponding to the current iteration, the immediate reward corresponding to the execution action in the state sequence in the experience pool is updated to obtain an updated immediate reward, wherein the updated immediate reward is used to influence the next iteration of the reinforcement learning model.
5. The method according to claim 4, characterized in that According to the global reward factor corresponding to the current iteration, the immediate reward corresponding to the execution action in the state sequence in the experience pool is updated, and the updated immediate reward is obtained by the following formula: Among them, the r t is the immediate reward corresponding to the execution action, and the r t+1 is the updated instant reward, is the global reward factor corresponding to this iteration, C_max is the maximum completion time of all processes of all pending orders corresponding to this iteration; Q is the preset reward function corresponding to the reinforcement learning model.
6. A job shop scheduling device, characterized in that: The device comprises: An acquisition module, used for acquiring operation information of a target workshop, wherein the operation information includes: information of each process of each pending order of the target workshop, and information of optional equipment for each process; An execution module is used to input the job information into the reinforcement learning model and execute an iteration step, wherein the iteration step includes: iteratively running the reinforcement learning model, executing multiple actions in each iteration of the reinforcement learning model, obtaining multiple environmental states and corresponding immediate rewards, and generating a corresponding state sequence each time an action is executed, and adding the state sequence to a preset experience pool, wherein the environmental state is: the process progress of the order to be processed and the equipment information selected for the process, and the state sequence includes: the environmental state before executing the action, the environmental state after executing the action, the immediate reward corresponding to the execution of the action, and the action corresponding to the execution of the action; the action is: determining the equipment information to be selected for the current process; After the current reinforcement learning model is iterated for a first preset number of times, the current ant colony algorithm model is run once, the input of the ant colony algorithm model is the job information, and the output is a plurality of production scheduling sequences of all processes in all pending orders; and the optimal production scheduling sequence is determined from the plurality of production scheduling sequences; After converting the optimal production scheduling sequence into the target state sequence of the reinforcement learning model, adding the target state sequence to the experience pool; Model updating step: determining whether the number of state sequences in the experience pool reaches a second preset number, and if so, updating the parameters of the reinforcement learning model based on the state sequences in the experience pool to obtain an updated reinforcement learning model; and updating the pheromone of the ant colony algorithm model after each operation of the ant colony algorithm model to obtain an updated ant colony algorithm model; Repeating the iterative step and the model updating step until the total number of iterations reaches a third preset number; Determine an optimal environment state from all state sequences of the reinforcement learning model, wherein the maximum completion time of all processes of all pending orders corresponding to the optimal environment state is the smallest; The job shop is scheduled based on the optimal environmental state.
7. The device according to claim 6, characterized in that The immediate reward corresponding to the execution action is determined by the pheromone concentrations between the path nodes corresponding to the environment state before the execution action and the environment state after the execution action in the ant colony algorithm model.
8. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the job shop scheduling method according to any one of claims 1 to 5 are implemented.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the job shop scheduling method according to any one of claims 1 to 5 are implemented.
10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the job shop scheduling method according to any one of claims 1 to 5.
Citation Information
Cited By
Cable production scheduling modeling method
CN119849325A
A cable production scheduling modeling method
CN119849325B
