Workshop scheduling optimization method and device, electronic equipment and storage medium

Abstract

The application provides a workshop scheduling optimization method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring current scheduling information of a workshop, extracting current local state information from the current scheduling information of the workshop; inputting the current local state information into a scheduling model corresponding to a current agent to obtain an optimal action set of the current agent; updating the current scheduling information of the workshop based on the optimal action set of the current agent to obtain next scheduling information of the workshop; inputting the next scheduling information of the workshop into a scheduling model corresponding to a next agent to obtain an optimal action set of the next agent; and determining a workshop scheduling scheme based on the optimal action sets corresponding to all agents of the workshop. The application solves the job shop scheduling problem through a multi-agent Markov decision process.

Description

Technical Field

[0001] This invention relates to the technical field of production workshop optimization, specifically to a workshop scheduling optimization method, apparatus, electronic device, and storage medium. Background Technology

[0002] The development of 5G and IoT (Internet of Things) technologies foreshadows a future where we can transmit massive amounts of data much faster at the terminal level. Consequently, smart factories are emerging. Simultaneously, with the growing trend towards product personalization, smart manufacturing facilities require automated production methods to adapt to the small-batch production paradigm of non-standard components. This approach demands flexibility, decoupling, and data-driven scheduling capabilities.

[0003] The Job Shop Scheduling Problem (JSP) serves as a relevant abstraction of the aforementioned production scenario. JSP is a well-known NP-hard (Non-deterministic Polynomial) problem. Various methods are employed to rationally arrange the processing sequence of each job on each machine to obtain an optimal solution close to our goal. However, an effective solution to the Job Shop Scheduling Problem remains lacking. Summary of the Invention

[0004] In view of this, it is necessary to provide a workshop scheduling optimization method, device, electronic equipment and storage medium to solve the workshop scheduling problem.

[0005] To address the above problems, this invention provides a workshop scheduling optimization method, comprising:

[0006] Obtain the current workshop scheduling information, and extract the current local state information from the current workshop scheduling information;

[0007] The current local state information is input into the scheduling model corresponding to the current agent to obtain the optimal action set of the current agent;

[0008] The current scheduling information of the workshop is updated based on the optimal action set of the current agent to obtain the next scheduling information of the workshop;

[0009] The next scheduling information of the workshop is input into the scheduling model corresponding to the next intelligent agent to obtain the optimal action set of the next intelligent agent;

[0010] Based on the optimal action set corresponding to all intelligent agents in the workshop, determine the workshop scheduling scheme;

[0011] The scheduling model corresponding to the agent is trained based on the following steps:

[0012] Obtain the global state information and local state information at the t-th digit from the historical scheduling information of the workshop; t≥1;

[0013] The t-th local state information is input into the Actor network corresponding to the t-th agent to obtain the t-th action set;

[0014] Generate the (t+1)th global state information and reward based on the t-th action set;

[0015] The t-th global state information is input into a preset value network to obtain the target value;

[0016] The target value and the reward are input into a preset target network to obtain the target network output value;

[0017] Based on the target value and the target network output value, update the value network and the Actor network corresponding to each agent to obtain the scheduling model corresponding to each agent.

[0018] Further, the step of inputting the t-th local state information into the Actor network corresponding to the t-th agent to obtain the t-th action set includes:

[0019] The t-th local state information is input into the Actor network corresponding to the t-th agent, and the t-th action set is determined based on a greedy strategy.

[0020] Furthermore, the formula for calculating the target network output value is as follows:

[0021]

[0022] Where y is the output value of the target network, R t R is the reward generated based on the t-th action set. t+1 V is the reward generated based on the (t+1)th action set, γ is a preset discount factor, and V critic (S t |θ critic ) represents the target value corresponding to the t-th global state information.

[0023] Further, updating the value network and the Actor network corresponding to each agent based on the target value and the target network output value includes:

[0024] Based on the target value and the target network output value, determine the first error loss corresponding to the value network and the second error loss corresponding to the Actor network;

[0025] The value network is updated based on the first error loss, and the Actor network corresponding to each agent is updated based on the second error loss.

[0026] Furthermore, the formula for calculating the first error loss is as follows:

[0027]

[0028] Among them, L c Let N be the first error loss, N be the number of batches used to train the scheduling model, y be the target network output value, and V be the value of the target network output. critic (S t |θ critic ) represents the target value corresponding to the t-th global state information.

[0029] Furthermore, the formula for calculating the second error loss is as follows:

[0030]

[0031] Among them, L a,j The second error loss is N, where N is the number of batches used to train the scheduling model, y is the target network output value, and V is the second error loss. critic (S t |θ critic S represents the target value corresponding to the t-th global state information. t For the t-th global state information, A j Let θ be the action set of the j-th agent. actor These are the network parameters of the Actor network.

[0032] Further, updating the Actor network corresponding to each agent based on the second error loss includes:

[0033] Based on the second error loss, and combined with the sampled policy gradient, the Actor network corresponding to each agent is updated.

[0034] The present invention also provides a workshop scheduling optimization device, comprising:

[0035] The extraction module is used to obtain the current scheduling information of the workshop and extract the current local state information from the current scheduling information of the workshop.

[0036] The first input module is used to input the current local state information into the scheduling model corresponding to the current agent to obtain the optimal action set of the current agent;

[0037] The update module is used to update the current workshop scheduling information based on the optimal action set of the current agent, so as to obtain the next workshop scheduling information;

[0038] The second input module is used to input the next scheduling information of the workshop into the scheduling model corresponding to the next intelligent agent to obtain the optimal action set of the next intelligent agent;

[0039] The scheme determination module is used to determine the workshop scheduling scheme based on the optimal action set corresponding to all intelligent agents in the workshop;

[0040] The scheduling model corresponding to the agent is trained based on the following steps:

[0041] Obtain the global state information and local state information at the t-th digit from the historical scheduling information of the workshop; t≥1;

[0042] The t-th local state information is input into the Actor network corresponding to the t-th agent to obtain the t-th action set;

[0043] Generate the (t+1)th global state information and reward based on the t-th action set;

[0044] The t-th global state information is input into a preset value network to obtain the target value;

[0045] The target value and the reward are input into a preset target network to obtain the target network output value;

[0046] Based on the target value and the target network output value, update the value network and the Actor network corresponding to each agent to obtain the scheduling model corresponding to each agent.

[0047] The present invention also provides an electronic device, including a memory and a processor, wherein,

[0048] The memory is used to store programs;

[0049] The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the workshop scheduling optimization method as described in any of the preceding claims.

[0050] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the workshop scheduling optimization method as described in any of the preceding claims.

[0051] The beneficial effects of the above implementation are as follows: The workshop scheduling optimization method, device, electronic device, and storage medium provided by the present invention obtain current workshop scheduling information, extract current local state information from the current workshop scheduling information; input the current local state information into the scheduling model corresponding to the current agent to obtain the optimal action set of the current agent; determine the workshop scheduling scheme based on the optimal action set of the current agent; the training steps of the scheduling model corresponding to the agent include: obtaining the t-th global state information and the t-th local state information from the historical workshop scheduling information; inputting the t-th local state information into the Actor network corresponding to the t-th agent to obtain the t-th action set; generating the t+1-th global state information and reward based on the t-th action set; inputting the t-th global state information into a preset value network to obtain the target value; inputting the target value and the reward into a preset target network to obtain the target network output value; updating the value network and the Actor network corresponding to each agent based on the target value and the target network output value to obtain the scheduling model corresponding to each agent.

[0052] This invention addresses the job shop scheduling problem using a multi-agent Markov decision process. In static environments, this algorithm outperforms traditional evolutionary algorithms and single-priority scheduling rules in achieving optimal solutions, particularly on large-scale datasets. Furthermore, due to its offline learning and online scheduling characteristics, it arrives at the optimal scheduling result faster than traditional evolutionary algorithms after model training.

[0053] Furthermore, the method provided by this invention can derive a superior scheduling solution compared to traditional evolutionary algorithms and priority scheduling rules for shop floor scheduling problems with random dynamic events, demonstrating the ability to flexibly handle dynamic events. The flexibility and adaptability of this algorithm model offer a compelling avenue for future research, showcasing the potential to develop more efficient and intelligent scheduling solutions for real-world manufacturing environments. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A flowchart of an embodiment of the workshop scheduling optimization method provided by the present invention;

[0056] Figure 2 This is a schematic diagram of a calculation example provided by the present invention;

[0057] Figure 3 A schematic diagram of an embodiment of the workshop scheduling optimization device provided by the present invention;

[0058] Figure 4 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0060] In the description of the embodiments of this application, unless otherwise stated, "a plurality of" means two or more.

[0061] In this embodiment of the invention, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, apparatus, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such process, method, product or device.

[0062] The naming or numbering of steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved.

[0063] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0064] This invention provides a workshop scheduling optimization method, apparatus, electronic device, and storage medium, which are described below.

[0065] like Figure 1 As shown, the present invention provides a workshop scheduling optimization method, comprising:

[0066] Step 110: Obtain the current workshop scheduling information, and extract the current local state information from the current workshop scheduling information;

[0067] Step 120: Input the current local state information into the scheduling model corresponding to the current agent to obtain the optimal action set of the current agent;

[0068] Step 130: Update the current workshop scheduling information based on the optimal action set of the current agent to obtain the next workshop scheduling information;

[0069] Step 140: Input the next scheduling information of the workshop into the scheduling model corresponding to the next intelligent agent to obtain the optimal action set of the next intelligent agent;

[0070] Step 150: Determine the workshop scheduling scheme based on the optimal action set corresponding to all intelligent agents in the workshop;

[0071] The scheduling model corresponding to the agent is trained based on the following steps:

[0072] Obtain the global state information and local state information at the t-th digit from the historical scheduling information of the workshop; t≥1;

[0073] The t-th local state information is input into the Actor network corresponding to the t-th agent to obtain the t-th action set;

[0074] Generate the (t+1)th global state information and reward based on the t-th action set;

[0075] The t-th global state information is input into a preset value network to obtain the target value;

[0076] The target value and the reward are input into a preset target network to obtain the target network output value;

[0077] Based on the target value and the target network output value, the value network and the Actor network corresponding to each agent are updated to obtain the scheduling model corresponding to each agent. It can be understood that the job shop scheduling problem (JSP) can be defined as follows: There are n jobs J = {J1, J2, J3, ..., J...} n} This needs to be done on m machines M = {M1, M2, M3, ..., M} m The processing is performed on a matching machine m; each step of the workpiece requires processing on a matching machine m. To better describe the problem described in this paper, some parameters and variables are defined as follows:

[0078] parameter:

[0079] n: number of workpieces

[0080] m: Number of machines

[0081] J i The i-th workpiece

[0082] M j The j-th machine

[0083] n i :workpiece J i Number of processes

[0084] n j Machine M j Number of workpieces to be processed

[0085] C: Completion Time

[0086] C i :workpiece J i Completion time

[0087] O ij :workpiece J i The process on machine j

[0088] p ij Process O ij Processing time

[0089] i: Job index, i = 1, 2, ..., n

[0090] j: Machine index, j = 1, 2, ..., m

[0091] k: Operation index, k = 1, 2, ..., m

[0092] Decision variables:

[0093]

[0094] Based on the symbols defined above, the JSP in this invention can be represented by the following mathematical model.

[0095] Minimize C max (1)

[0096]

[0097] Equation (1) is the objective function, indicating that the goal of the problem is to minimize the maximum completion time. Equation (2) represents the basic constraints of the problem, where (a) represents the completion time constraint, that is, each workpiece must complete all operations within a specified time; (b) and (c) are the workpiece constraint and machine constraint, respectively, that is, the machine M at the same time... j Only one workpiece can be processed at a time. Finally, (d) is the processing sequence constraint, indicating that for each workpiece J... i Processing must be carried out in the predetermined order of operations.

[0098] This invention addresses the JSP problem by enhancing the Actor-Critic model. In this invention, each machine can be viewed as a separate agent, with each agent having its own Actor network to guide its operations (i.e., selecting jobs to process). Furthermore, this invention introduces a shared Critic network to guide the updates of the individual Actor networks. Compared to the original one-to-one Actor-Critic model, this invention is more like a teacher guiding a group of students, with one teacher (SharedCritic) guiding multiple students (Actors) towards a common goal. To capture both global and local information, this invention proposes a state feature representation that combines both. Global information is input into the Critic network, while local information is input into the corresponding Actor network. This algorithmic structure allows for almost complete decoupling during execution and provides the flexibility needed to effectively handle random dynamic events. Furthermore, this invention designs a single-step reward function to guide network updates to maximize average machine utilization.

[0099] (1) State representation

[0100] This invention proposes a new state representation, as shown in Table 1, wherein the global state contains global information of the job shop to guide the shared Critic network, and the local state represents local information in each machine (for each machine as an agent) to guide the Actor network of each agent.

[0101] Table 1: State Representation

[0102]

[0103] (2) Action space

[0104] The agent designed in this invention performs actions by selecting the highest-priority job for processing according to scheduling rules. To fully utilize the agent's ability to learn from experience, this invention selects multiple scheduling rules related to machine utilization as the scheduling space, as shown in Table 2. By merging multiple scheduling rules in the scheduling space, the agent can explore different strategies and learn from interactions with the environment. This approach allows the agent to adjust its operations based on observed performance and encourages the discovery of effective scheduling strategies that optimize machine utilization.

[0105] Table 2: Scheduling Rules

[0106]

[0107] (3) Reward function

[0108] Due to the dynamic nature of the job shop scheduling problem, timely feedback is crucial for agents to adjust their strategies and adapt to changing conditions. This invention employs single-step rewards to provide immediate feedback. By using single-step rewards based on machine utilization at each time step, the agent can quickly understand which decisions positively impact average machine utilization. Higher average machine utilization indicates a shorter maximum completion time. When an agent's actions contribute to improving average machine utilization, it results in positive feedback from the gradient direction perspective. In reinforcement learning, the goal is to maximize the expected cumulative reward over time. The reward signal guides the agent on whether its actions are beneficial, as shown in Table 3.

[0109] Table 3: Reward Signals

[0110]

[0111] (4) Algorithm Execution Flow

[0112] This algorithm combines the advantages of "offline learning and online scheduling," enabling centralized training and distributed execution. Each machine represents a separate agent. During online scheduling, only local information about each machine needs to be input to determine its action. This method allows for complete decoupling of the scheduling process between machines when multiple agents are pursuing a common goal.

[0113] It should be noted that the training process of this algorithm (offline training) is centralized, meaning that the Critic network parameters and the examples (S, A, R, S) in the experience pool are shared among all agents. However, during execution (online scheduling), each agent acts independently and does not communicate directly with other agents, achieving distributed execution.

[0114] 1. Execution process:

[0115] Starting from the current workshop scheduling information, local state information is extracted for each agent (machine). During single-step scheduling, the local state information is input into the offline-trained model, and the output is the optimal action for each agent (greedy strategy: a certain probability of obtaining a random action). The action set of all agents and the input are fed into the environment to obtain the next state, updating the workshop scheduling information as the input for the next loop, until scheduling is completed. The entire process is executed E times, and the optimal solution is taken as the scheduling result.

[0116] 2. Training process:

[0117] Initialize the experience pool capacity and network parameters;

[0118] Obtain initial global and local state information from the workshop scheduling environment;

[0119] For each agent (machine), local information is input to obtain the action set under the current network parameters, using a greedy strategy;

[0120] Storage (S) t ,A,R t ,S t+1 Tuple information is added to the experience pool;

[0121] Samples are drawn in batches from the experience pool and input into the target network (the initial parameters of the target network are the same as those of the Critic network, and there is only one target network);

[0122] The target value is calculated using the Bellman equation by combining the target network output value.

[0123] Based on the target value, the loss is calculated using the MSE (Mean-Square Error) equation, and the gradient descent strategy is used to optimize the network parameters based on the loss.

[0124] For each agent's Actor network, based on the value output by the Critic network, the error loss is calculated according to the formula, and the policy gradient is optimized to ensure that the gradient optimization direction maximizes the value of the output action.

[0125] The parameters of the target network are updated every C steps (the update method is to copy the Critic network parameters);

[0126] The environment information is updated based on the action as the input for the next loop, and the entire training process iterates E times to obtain the trained algorithm model until the scheduling ends.

[0127] In some embodiments, the step of inputting the t-th local state information into the Actor network corresponding to the t-th agent to obtain the t-th action set includes:

[0128] The t-th local state information is input into the Actor network corresponding to the t-th agent, and the t-th action set is determined based on a greedy strategy.

[0129] It is understandable that for each intelligent agent (machine), local information is input to obtain the action set under the current network parameters, and a greedy strategy is used. A greedy strategy is a problem-solving method that starts from the initial state of the problem and obtains the optimal value (or a better solution) through several greedy choices.

[0130] In some embodiments, the formula for calculating the target network output value is:

[0131]

[0132] Where y is the output value of the target network, R tR is the reward generated based on the t-th action set. t+1 V is the reward generated based on the (t+1)th action set, γ is a preset discount factor, and V critic (S t |θ critic ) represents the target value corresponding to the t-th global state information.

[0133] In some embodiments, updating the value network and the Actor network corresponding to each agent based on the target value and the target network output value includes:

[0134] Based on the target value and the target network output value, determine the first error loss corresponding to the value network and the second error loss corresponding to the Actor network;

[0135] The value network is updated based on the first error loss, and the Actor network corresponding to each agent is updated based on the second error loss.

[0136] In some embodiments, the formula for calculating the first error loss is:

[0137]

[0138] Among them, L c Let N be the first error loss, N be the number of batches used to train the scheduling model, y be the target network output value, and V be the value of the target network output. critic (S t |θ critic ) represents the target value corresponding to the t-th global state information.

[0139] In some embodiments, the formula for calculating the second error loss is:

[0140]

[0141] Among them, L a,j The second error loss is N, where N is the number of batches used to train the scheduling model, y is the target network output value, and V is the second error loss. critic (S t |θ critic S represents the target value corresponding to the t-th global state information. t For the t-th global state information, A j Let θ be the action set of the j-th agent. actor These are the network parameters of the Actor network.

[0142] In some embodiments, updating the Actor network corresponding to each agent based on the second error loss includes:

[0143] Based on the second error loss, and combined with the sampled policy gradient, the Actor network corresponding to each agent is updated.

[0144] In other embodiments, examples are demonstrated:

[0145] We selected publicly available datasets from the OR-library to verify the algorithm's performance. The OR-library is a widely used benchmark dataset for various optimization problems, including the job shop scheduling problem.

[0146] parameter:

[0147] To ensure reproducibility and provide insight into the experimental setup, we present the hyperparameter settings used in our study. The following table summarizes the hyperparameters used to train the deep reinforcement learning algorithm for the job shop scheduling problem:

[0148] The algorithm implementation steps include offline learning and online scheduling, among which,

[0149] Offline learning:

[0150] The workshop scheduling reinforcement learning environment acquires historical production data from the actual production environment and extracts initial global information S. t and local information S t,j ;

[0151] The agent interacts with the environment and returns a set of actions A to the environment, which then generates the next state S. t+1 and reward R t Update the agent;

[0152] Storage Quadruple t ,A,R t ,S t+1 >In the experience pool (if the experience pool is full, replace the experience pool with the longest storage time);

[0153] The enhanced Actor-Critic model samples N batches of tuples from the experience pool. t ,A,R t ,S t+1 >;

[0154] Enter S t Go to the Critic network and obtain value V critic (S t |θ critic );

[0155] Enter S t+1 The target value is obtained by going to the target network and calculating it using the following formula: Where γ is the discount factor;

[0156] ​​Update the Critic network by minimizing the mean square Bellman error:

[0157] Update the Actor network of the current agent using the sampled policy gradient:

[0158] Switch to the next agent and repeat the steps of updating the current agent's Actor network using the sampled policy gradient until all agents have been trained.

[0159] Update the parameters of the target network every K steps;

[0160] The number of iterations E continues until training is complete.

[0161] Online dispatching:

[0162] The workshop scheduling reinforcement learning environment acquires real-time production data from the actual production environment and extracts initial global information S. t and local information S t,j ;

[0163] The agent interacts with the environment and returns an action set A and the next state S. t+1 and scheduling completion flag F;

[0164] Repeat the steps of the agent interacting with the environment and returning the action set, the next state, and the scheduling completion flag until scheduling is complete (i.e., F←true).

[0165] Taking the public dataset "ft06" of Fish in OR-Library as an example, as shown in Table 4:

[0166] Table 4: FT06 Case Study

[0167]

[0168] Taking row 0 as an example, it indicates that the processing sequence of the 0th workpiece on the machine is machine 2-0-1-3-5-4, and the processing time is 1-3-6-7-3-6 respectively. Figure 2 As shown.

[0169] Input the algorithm example into the reinforcement learning environment simulation and extract the initial state:

[0170] S t ={U ave U std CO, CJ ave CJ std}

[0171] S t,j ={U j RPT j,RO j}

[0172] Initial global information S t and local information state S t,j The input is given to the intelligent agent, namely our proposed enhanced Actor-Critic algorithm model, which undergoes an "offline learning" training process and then outputs the results.

[0173] Then, the trained model is used to perform an "online scheduling" process in a real environment (taking FT06 as an example). During the simulation of the interaction between the environment and the intelligent agent, the processing sequence of each machine's process is recorded, a Gantt chart is drawn, and the final required scheduling scheme is obtained.

[0174] In summary, this invention proposes an improved Actor-Critic algorithm model that uses a multi-agent Markov decision process (MADP) to solve the job shop scheduling problem. In static environments, this algorithm achieves better solutions more easily than traditional evolutionary algorithms and single-priority scheduling rules, especially on large-scale datasets; furthermore, due to its offline learning and online scheduling characteristics, it arrives at the optimal scheduling result faster than traditional evolutionary algorithms after model training.

[0175] Furthermore, the method provided by this invention can derive a superior scheduling solution compared to traditional evolutionary algorithms and priority scheduling rules for shop floor scheduling problems with random dynamic events, demonstrating the ability to flexibly handle dynamic events. The flexibility and adaptability of this algorithm model offer a compelling avenue for future research, showcasing the potential to develop more efficient and intelligent scheduling solutions for real-world manufacturing environments.

[0176] like Figure 3 As shown, the present invention also provides a workshop scheduling optimization device 300, comprising:

[0177] Extraction module 310 is used to obtain current workshop scheduling information and extract current local state information from the current workshop scheduling information;

[0178] The first input module 320 is used to input the current local state information into the scheduling model corresponding to the current agent to obtain the optimal action set of the current agent;

[0179] The update module 330 is used to update the current workshop scheduling information based on the optimal action set of the current intelligent agent to obtain the next workshop scheduling information;

[0180] The second input module 340 is used to input the next scheduling information of the workshop into the scheduling model corresponding to the next intelligent agent to obtain the optimal action set of the next intelligent agent;

[0181] The scheme determination module 350 is used to determine the workshop scheduling scheme based on the optimal action set corresponding to all intelligent agents in the workshop;

[0182] The scheduling model corresponding to the agent is trained based on the following steps:

[0183] Obtain the global state information and local state information at the t-th digit from the historical scheduling information of the workshop; t≥1;

[0184] The t-th local state information is input into the Actor network corresponding to the t-th agent to obtain the t-th action set;

[0185] Generate the (t+1)th global state information and reward based on the t-th action set;

[0186] The t-th global state information is input into a preset value network to obtain the target value;

[0187] The target value and the reward are input into a preset target network to obtain the target network output value;

[0188] Based on the target value and the target network output value, update the value network and the Actor network corresponding to each agent to obtain the scheduling model corresponding to each agent.

[0189] The workshop scheduling optimization device provided in the above embodiments can realize the technical solutions described in the above workshop scheduling optimization method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above workshop scheduling optimization method embodiments, which will not be repeated here.

[0190] like Figure 4 As shown, the present invention also provides an electronic device 400. The electronic device 400 includes a processor 401, a memory 402, and a display 403. Figure 4 Only some components of the electronic device 400 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0191] In some embodiments, memory 402 may be an internal storage unit of electronic device 400, such as a hard disk or memory of electronic device 400. In other embodiments, memory 402 may also be an external storage device of electronic device 400, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 400.

[0192] Furthermore, the memory 402 may include both internal storage units of the electronic device 400 and external storage devices. The memory 402 is used to store application software and various types of data installed on the electronic device 400.

[0193] In some embodiments, processor 401 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 402 or process data, such as the workshop scheduling optimization method of the present invention.

[0194] In some embodiments, display 403 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 403 is used to display information from electronic device 400 and to display a visual user interface. Components 401-403 of electronic device 400 communicate with each other via a system bus.

[0195] In some embodiments of the present invention, when the processor 401 executes the shop floor scheduling optimization program in the memory 402, the following steps can be implemented:

[0196] Obtain the current workshop scheduling information, and extract the current local state information from the current workshop scheduling information;

[0197] The current local state information is input into the scheduling model corresponding to the current agent to obtain the optimal action set of the current agent;

[0198] The current scheduling information of the workshop is updated based on the optimal action set of the current agent to obtain the next scheduling information of the workshop;

[0199] The next scheduling information of the workshop is input into the scheduling model corresponding to the next intelligent agent to obtain the optimal action set of the next intelligent agent;

[0200] Based on the optimal action set corresponding to all intelligent agents in the workshop, determine the workshop scheduling scheme;

[0201] The scheduling model corresponding to the agent is trained based on the following steps:

[0202] Obtain the global state information and local state information at the t-th digit from the historical scheduling information of the workshop; t≥1;

[0203] The t-th local state information is input into the Actor network corresponding to the t-th agent to obtain the t-th action set;

[0204] Generate the (t+1)th global state information and reward based on the t-th action set;

[0205] The t-th global state information is input into a preset value network to obtain the target value;

[0206] The target value and the reward are input into a preset target network to obtain the target network output value;

[0207] Based on the target value and the target network output value, update the value network and the Actor network corresponding to each agent to obtain the scheduling model corresponding to each agent.

[0208] It should be understood that when the processor 401 executes the shop floor scheduling optimization program in the memory 402, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0209] Furthermore, the embodiments of the present invention do not specifically limit the type of electronic device 400 mentioned. Electronic device 400 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, electronic device 400 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0210] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the shop floor scheduling optimization method provided by the methods described above, the method comprising:

[0211] Obtain the current workshop scheduling information, and extract the current local state information from the current workshop scheduling information;

[0212] The current local state information is input into the scheduling model corresponding to the current agent to obtain the optimal action set of the current agent;

[0213] The current scheduling information of the workshop is updated based on the optimal action set of the current agent to obtain the next scheduling information of the workshop;

[0214] The next scheduling information of the workshop is input into the scheduling model corresponding to the next intelligent agent to obtain the optimal action set of the next intelligent agent;

[0215] Based on the optimal action set corresponding to all intelligent agents in the workshop, determine the workshop scheduling scheme;

[0216] The scheduling model corresponding to the agent is trained based on the following steps:

[0217] Obtain the global state information and local state information at the t-th digit from the historical scheduling information of the workshop; t≥1;

[0218] The t-th local state information is input into the Actor network corresponding to the t-th agent to obtain the t-th action set;

[0219] Generate the (t+1)th global state information and reward based on the t-th action set;

[0220] The t-th global state information is input into a preset value network to obtain the target value;

[0221] The target value and the reward are input into a preset target network to obtain the target network output value;

[0222] Based on the target value and the target network output value, update the value network and the Actor network corresponding to each agent to obtain the scheduling model corresponding to each agent.

[0223] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0224] The workshop scheduling optimization method, apparatus, electronic device, and storage medium provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A workshop scheduling optimization method, characterized in that, include: Obtain the current workshop scheduling information, and extract the current local state information from the current workshop scheduling information; The current local state information is input into the scheduling model corresponding to the current agent to obtain the optimal action set of the current agent; The current scheduling information of the workshop is updated based on the optimal action set of the current agent to obtain the next scheduling information of the workshop; The next scheduling information of the workshop is input into the scheduling model corresponding to the next intelligent agent to obtain the optimal action set of the next intelligent agent; Based on the optimal action set corresponding to all intelligent agents in the workshop, a workshop scheduling scheme is determined. The scheduling model corresponding to the agent is trained based on the following steps: Obtain the global state information and local state information at the t-th digit from the historical scheduling information of the workshop; t≥1; The t-th local state information is input into the Actor network corresponding to the t-th agent to obtain the t-th action set; Generate the (t+1)th global state information and reward based on the t-th action set; The t-th global state information is input into a preset value network to obtain the target value; The target value and the reward are input into a preset target network to obtain the target network output value; Based on the target value and the target network output value, update the value network and the Actor network corresponding to each agent to obtain the scheduling model corresponding to each agent.

2. The workshop scheduling optimization method according to claim 1, characterized in that, The step of inputting the t-th local state information into the Actor network corresponding to the t-th agent to obtain the t-th action set includes: The t-th local state information is input into the Actor network corresponding to the t-th agent, and the t-th action set is determined based on a greedy strategy.

3. The workshop scheduling optimization method according to claim 1, characterized in that, The formula for calculating the target network output value is: in, y The output value for the target network. R t The reward generated based on the t-th action set, R t+1 The reward is generated based on the (t+1)th action set. γ This is the preset discount factor.

4. The workshop scheduling optimization method according to claim 1, characterized in that, The step of updating the value network and the Actor network corresponding to each agent based on the target value and the target network output value includes: Based on the target value and the target network output value, determine the first error loss corresponding to the value network and the second error loss corresponding to the Actor network; The value network is updated based on the first error loss, and the Actor network corresponding to each agent is updated based on the second error loss.

5. The workshop scheduling optimization method according to claim 4, characterized in that, The formula for calculating the first error loss is: in, L c For the first error loss, N The number of batches used to train the scheduling model. y The output value for the target network. The target value corresponding to the t-th global state information. S t This represents the global state information for the t-th time.

6. The workshop scheduling optimization method according to claim 4, characterized in that, The formula for calculating the second error loss is: in, L a,j This is the second error loss. N The number of batches used to train the scheduling model. y The output value for the target network. The target value corresponding to the t-th global state information. S t For the t-th global state information, A j For the first j A set of actions for an intelligent agent. θ actor These are the network parameters of the Actor network.

7. The workshop scheduling optimization method according to claim 4, characterized in that, The step of updating the Actor network corresponding to each agent based on the second error loss includes: Based on the second error loss, and combined with the sampled policy gradient, the Actor network corresponding to each agent is updated.

8. A workshop scheduling optimization device, characterized in that, include: The extraction module is used to obtain the current scheduling information of the workshop and extract the current local state information from the current scheduling information of the workshop. The first input module is used to input the current local state information into the scheduling model corresponding to the current agent to obtain the optimal action set of the current agent; The update module is used to update the current workshop scheduling information based on the optimal action set of the current agent, so as to obtain the next workshop scheduling information; The second input module is used to input the next scheduling information of the workshop into the scheduling model corresponding to the next intelligent agent to obtain the optimal action set of the next intelligent agent; The scheme determination module is used to determine the workshop scheduling scheme based on the optimal action set corresponding to all intelligent agents in the workshop; The scheduling model corresponding to the agent is trained based on the following steps: Obtain the global state information and local state information at the t-th digit from the historical scheduling information of the workshop; t≥1; The t-th local state information is input into the Actor network corresponding to the t-th agent to obtain the t-th action set; Generate the (t+1)th global state information and reward based on the t-th action set; The t-th global state information is input into a preset value network to obtain the target value; The target value and the reward are input into a preset target network to obtain the target network output value; Based on the target value and the target network output value, update the value network and the Actor network corresponding to each agent to obtain the scheduling model corresponding to each agent.

9. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the workshop scheduling optimization method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the workshop scheduling optimization method as described in any one of claims 1 to 7.

Images