Production scheduling device, production scheduling method and production scheduling program

Abstract

To enable an efficient scheduling using reinforcement learning.SOLUTION: A production scheduling device that makes a production schedule based on an activity selection model includes: an experience storing unit which selects an activity in the production schedule, and which stores, as tree-structure information representing a state transition, first experience data including the selected activity; an experience updating unit that re-calculates a reward using a reward function and past experience data when the reward function is changed; and a learning unit that updates the parameter of the activity selection model using second experience data including the updated reward by the experience updating unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a production scheduling device, a production scheduling method, and a production scheduling program. [Background technology]

[0002] Conventionally, scheduling at production sites is carried out to determine the order in which each product is manufactured by each machine (hereinafter, also referred to as a schedule) in order to increase productivity while strictly adhering to delivery deadlines.

[0003] In such scheduling at production sites, the total number of possible production sequences becomes enormous, and it may not be possible to search all of them in real time. Therefore, there is generally no efficient algorithm for finding the true optimal solution, and metaheuristic methods are used as a search technology for approximate solutions that search for the best possible solution.

[0004] On the other hand, metaheuristic methods also require a separate search for scheduling every time there is a change in inventory, orders, etc., which reduces the efficiency of scheduling work.

[0005] In recent years, reinforcement learning has been used as a scheduling method in production sites (Patent Document 1). Reinforcement learning depends on the definition of a reward function that indicates the reward, and has been learned for the purpose of single-objective optimization, such as minimizing the time it takes to complete the production of a product. However, in actual production sites, reinforcement learning needs to learn for the purpose of multi-objective optimization, such as occupational safety and health and business planning, in addition to minimizing the time it takes to complete the production of a product. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2022-135909 Summary of the Invention [Problem to be solved by the invention]

[0007] However, with the above-mentioned conventional techniques, it may be difficult to efficiently perform scheduling using reinforcement learning.

[0008] Generally, reinforcement learning relies on the definition of a reward function that indicates the reward, and learning is performed for the purpose of single-objective optimization, such as minimizing the time it takes to complete the production of a product. However, in actual production sites, reinforcement learning needs to learn for the purpose of multi-objective optimization, such as occupational safety and health and business planning, in addition to minimizing the time it takes to complete the production of a product.

[0009] For example, when learning for the purpose of multi-objective optimization, it is difficult to define the reward in advance, and humans must adjust the reward function based on the schedule output by reinforcement learning. However, conventionally, adjusting the value of the reward function requires redoing learning, including the state, action, and state after action, in addition to the changed reward function, which takes a significant amount of time. [Means for solving the problem]

[0010] A production scheduling device according to the present invention is a production scheduling device that formulates a production schedule based on a behavior selection model, and is characterized by comprising: an experience memory unit that selects actions in the production schedule and stores first experience data including the selected actions as tree-structured information representing state transitions; an experience update unit that, when a reward function is changed, recalculates the reward using the reward function and past experience data; and a learning unit that updates the parameters of the behavior selection model using second experience data including the reward updated by the experience update unit.

[0011] Furthermore, a production scheduling method according to the present invention is a production scheduling method for formulating a production schedule based on a behavior selection model, and is characterized by including: an experience storing step of selecting an action in the production schedule and storing first experience data including the selected action as tree-structured information representing state transitions; an experience updating step of recalculating the reward using the reward function and past experience data when the reward function is changed; and a learning step of updating the parameters of the behavior selection model using second experience data including the reward updated by the experience updating step.

[0012] Furthermore, a production scheduling program according to the present invention is a production scheduling program that creates a production schedule based on a behavior selection model, and is characterized in that it causes a computer to execute the following steps: an experience storing step that selects an action in the production schedule and stores first experience data including the selected action as tree-structured information represented by state transitions; an experience updating step that, when a reward function is changed, recalculates the reward using the reward function and past experience data; and a learning step that updates the parameters of the behavior selection model using second experience data including the reward updated by the experience updating step. [Effects of the Invention]

[0013] According to the present invention, it is possible to effectively perform scheduling using reinforcement learning. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a production scheduling device according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of setting information according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of setting information according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of input information according to the embodiment. [Figure 5]FIG. 5 is a diagram illustrating an example of inventory information according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of an experience update process according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a Gantt chart according to the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of the production scheduling device according to the embodiment. [Figure 9] FIG. 9 is a flowchart illustrating an example of a production scheduling device according to an embodiment. [Figure 10] FIG. 10 is a hardware configuration diagram showing an example of a computer that realizes the functions of the production scheduling device. DETAILED DESCRIPTION OF THE INVENTION

[0015] Below, a detailed description will be given of a production scheduling device, a production scheduling method, and a production scheduling program (hereinafter referred to as "embodiments") according to the present application, with reference to the drawings. Note that the scheduling device, the scheduling method, and the scheduling program according to the present application are not limited to these embodiments. Furthermore, the same components in the following embodiments are given the same reference numerals, and duplicated descriptions will be omitted.

[0016] 1. Introduction Production scheduling devices are used to determine the manufacturing sequence of products such as chemicals, pharmaceuticals, or food. In a production scheduling device, an episode is defined as the period from the start of scheduling to the end of scheduling. For example, an episode ends when work allocation to fulfill all orders from the start of scheduling is completed, or when work allocation for a specified planning period is completed.

[0017] The production scheduling device performs scheduling using reinforcement learning. For example, in reinforcement learning, the environment outputs an observation for a state. Next, the agent outputs an action for the observation. Subsequently, the environment transitions the state in response to the action from the agent (hereinafter, this may be referred to as a state transition). The environment also outputs a reward for the state and the action. Then, the agent improves the policy in response to the reward (hereinafter, this may be referred to as learning). Here, the agent has a behavior selection model that inputs the state and selects an action to output.

[0018] Here, the production scheduling device creates a production schedule (hereinafter also referred to as a schedule) based on a behavior selection model. The production scheduling device saves experience data including states, actions, states after actions, and rewards obtained by searching the behavior selection model as tree-structured information representing state transitions. The production scheduling device then updates only the rewards of the experience data and reuses the other experience data. In other words, by updating only the rewards, which are a part of the experience data, the production scheduling device can reduce the calculation load and make the search for the behavior selection model more efficient, thereby enabling efficient scheduling using reinforcement learning.

[0019] Next, an example of input to the production scheduling device in order to output a schedule will be described. For example, the production scheduling device receives as input setting information (hereinafter also referred to as invariant information), which is information that is fixed as operational requirements such as master data specific to the factory, input information (hereinafter also referred to as variable information), which is information about time slices of factory operations, a compensation adjustment parameter, and an operation mode. Specifically, the production scheduling device receives as input setting information, which is invariant information such as machine configuration, input information, which is variable information such as inventory, a compensation adjustment parameter, and an operation mode.

[0020] Next, an example of an operation mode will be described. The production scheduling device can select between a development stage and an operation stage as its operation mode. For example, the development stage is a mode in which reinforcement learning is used to train a schedule using multiple pieces of input information as input. On the other hand, the operation stage is a mode in which a Gantt chart showing the schedule is output using a single piece of input information as input.

[0021] Next, an example of the reward adjustment parameter will be described. Here, the reward adjustment parameter is a hyperparameter of a reward function that indicates the reward, and the production scheduling device may receive the reward adjustment parameter or a reward function as the reward adjustment parameter from the user. Here, the production scheduling device can indicate the reward adjustment parameter using objectives and constraints.

[0022] For example, the objectives of a production scheduling device are to shorten the time it takes to complete the production of all products ordered by a user (hereinafter also referred to as makespan), to improve production efficiency, profits, occupational health and safety, business continuity, and equipment utilization rate. Also, the constraints of a production scheduling device include when the delivery date requested by the user is delayed and when inventory meets upper and lower limits. Note that a production scheduling device may include either one or both of the objectives and constraints.

[0023] Here, the production scheduling device defines either one or both of the above objectives and constraints as rewards. For example, the production scheduling device defines rewards according to objectives such as production efficiency or profit efficiency (hereinafter also referred to as evaluation indices) and constraints such as delays in delivery dates (hereinafter also referred to as penalty indices). Furthermore, when the production scheduling device receives from the user the development stage or operation stage as the operation mode, it may learn the behavior selection model by receiving from the user a change in the reward function that indicates rewards according to either one or both of the objectives and constraints.

[0024] More specifically, the production scheduling device expresses the reward function as a weighted sum of an evaluation index and a penalty index. For example, the production scheduling device accepts the value of the reward function from the user in order to adjust the value of the reward function according to the objectives and constraints, and performs scheduling using a behavior selection model. Note that when learning the behavior selection model, the production scheduling device may perform iterative learning by accepting appropriate changes or adjustments to the reward function from the user.

[0025] Next, we will explain an example of a schedule output by the production scheduling device. For example, the production scheduling device outputs a schedule that includes the timing of ordering materials required to manufacture a product, the selection of machines, and the timing of manufacturing.

[0026] 2. Configuration of the production scheduling device Next, the configuration of the production scheduling device 100 will be explained using Figure 1. Figure 1 is a diagram showing an example of the configuration of a production scheduling device according to an embodiment. As shown in Figure 1, the production scheduling device 100 has a communication unit 110, a storage unit 120, and a control unit 130. The production scheduling device 100 also creates a production schedule based on a behavior selection model. The processing of each unit of the production scheduling device 100 will be explained below.

[0027] The communication unit 110 enables communication between an external device and the control unit 130. The communication unit 110 is realized by a NIC (Network Interface Card) or the like, and enables communication between the external device and the control unit 130 via a telecommunication line such as a LAN (Local Area Network) or the Internet.

[0028] The storage unit 120 stores data and programs necessary for various processes performed by the control unit 130. For example, as shown in Fig. 1, the storage unit 120 stores a behavior selection model storage unit 121, an experience storage unit 122, and a state storage unit 123. The storage unit 120 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0029] The behavior selection model storage unit 121 stores a behavior selection model. For example, the behavior selection model storage unit 121 stores a behavior selection model that outputs a behavior in response to a state input. Furthermore, when the driving mode is in the operation stage, the behavior selection model storage unit 121 reads out parameters of the behavior selection model specified by the user and stores parameters of the learned behavior selection model so that a behavior selection model can be constructed. Note that, as the behavior selection model, a model generated by learning using a neural network and the ε (epsilon)-greedy method that shows stochastic behavior can be selected.

[0030] Here, the behavior selection model storage unit 121 may store a behavior selection model that outputs a product ID as a behavior when the input state is the product to be worked on.

[0031] For example, if the environment unit 132 is designed to output a list of candidate actions (also called a candidate action list) for the current operation status and inventory status, the behavior selection model may be a mechanism that takes the current operation status, inventory status, and candidate actions as input, infers an action value for each candidate action, and selects an action that maximizes the action value. Alternatively, if the environment unit 132 is designed to output a list of operation statuses and inventory statuses after a state transition based on candidate actions for the current operation status and inventory status (also called a candidate state list), the behavior selection model may be a mechanism that takes the candidate state as input, infers a state value for each candidate state, and selects an action that maximizes the state value.

[0032] In addition, the behavior selection model storage unit 121 may store a behavior selection model that directly infers and outputs behavior using multiple states, behaviors, and reward trajectories as input, or it may store a behavior selection model that outputs the value or behavior trajectory for each state.

[0033] The experience storage unit 122 selects an action in the production schedule and stores the experience data (first experience data) including the selected action as tree-structured information representing a state transition. For example, the experience storage unit 122 stores, as the experience data, the operation state before the state transition, the inventory state, the action, the operation state after the state transition due to the action, the inventory state, the reward, whether the episode has ended, and the input information ID. Specifically, the experience data is stored as tree-structured information of "state, action, state after action, and reward" in a series of time steps in which an agent selects an action in a state, the environment transitions to the next state that reflects the action, and the merits or demerits of the action are evaluated as a reward.

[0034] Furthermore, for example, the experience storage unit 122 stores experience data updated by the environment unit 132 (third experience data).

[0035] The state storage unit 123 stores states in reinforcement learning. For example, the state storage unit 123 stores setting information and input information as states in reinforcement learning. Specifically, the setting information includes master data of fixed information specific to the factory, such as the configuration of processes and machines and the production capacity of each machine. The input information includes operating status, inventory status, and order information such as product type, quantity, and delivery date for the scheduling period. The operating status stores the operating status of each machine, and the inventory status stores the inventory amount of each product.

[0036] Next, the control unit 130 is realized using a CPU (Central Processing Unit), an NP (Network Processor), an FPGA (Field Programmable Gate Array), or the like, and executes a processing program stored in memory. As shown in Fig. 1, the control unit 130 also has a reception unit 131, an environment unit 132, an experience update unit 133, a learning unit 134, and an output unit 135. Each unit of the control unit 130 will be described below.

[0037] The reception unit 131 receives input of setting information, input information, remuneration adjustment parameters, and operation modes from a user. For example, the setting information includes input regarding the machine configuration, connected machines, the number of days required for each machine in each process depending on the type of product, and the processing quantity of the machine. The input information includes the inventory status at the beginning of the period, the operating status at the beginning of the period, and order information linked to the input information ID.

[0038] Furthermore, the reward adjustment parameter is a hyperparameter of a reward function that indicates the reward, and the receiving unit 131 may receive the reward function. Specifically, the receiving unit 131 receives a reward adjustment parameter according to the objectives and constraints of production scheduling set by the user. The receiving unit 131 then receives input of the development stage or the operation stage as the operation mode. Note that the receiving unit 131 may receive some of the setting information, input information, reward adjustment parameter, and operation mode as input.

[0039] The environment unit 132 selects an action in the production schedule from the action selection model before the update, compares third experience data including the selected action with second experience data including the past experience data and the updated reward, and updates the second experience data with the third experience data if the third experience data is not included.

[0040] Specifically, the behavior selection model selects a behavior based on a state observed by the environment unit 132, which includes setting information and input information received from the user by the reception unit 131. Next, the environment unit 132 compares the third empirical data including the behavior selected in the behavior selection model with the second empirical data in terms of the input information ID, operating status, and inventory status, and updates the third empirical data if they do not match.

[0041] Here, for example, the behavior selection model may select a different behavior when the reward is updated to improve the policy relative to the reward. Therefore, the environment unit 132 outputs empirical data (third empirical data) that is different from the empirical data (second empirical data). Note that the second empirical data and the third empirical data may be similar.

[0042] Next, an example of the role of the environment unit 132 in reinforcement learning will be described. Based on the setting information, input information, reward adjustment parameters, and operation mode received from the user by the receiving unit 131, and the action obtained from the action selection model, the environment unit 132 outputs experience data including a state before a state transition (operation state, inventory state), an action or a state after a state transition due to an action (operation state, inventory state), a reward, whether or not an episode has ended, and an input information ID. Here, the environment unit 132 updates the experience data by transitioning states based on the results output by the action selection model. The environment unit 132 also determines whether an episode has ended, for example, when work allocation to satisfy all orders has been completed or when work allocation for a predetermined planning period has been completed. The environment unit 132 also calculates a reward using the operation state and inventory state as input. The environment unit 132 may calculate a reward only for an episode end state, and set the reward to "0" if the episode has not ended.

[0043] The environment unit 132 also performs initialization when production scheduling starts. The environment unit 132 also performs initialization when an operating status or inventory status is received. Here, if the operation mode is in the development stage and multiple pieces of input information are input, the environment unit 132 replaces the input information used for initialization for each episode. The initialization performed by the environment unit 132 is not limited to initialization of the operating status or inventory status, and may include setting up a behavior selection model, or may include setting up a behavior selection model and past experience data without initializing the operating status or inventory status.

[0044] When the reward function is changed, the experience updating unit 133 recalculates the reward using the new reward function and past experience data. For example, when the reward function is changed, the experience updating unit 133 reuses the state, action, and state after action from the past experience data, and changes only the changed reward function to recalculate the reward. Specifically, the experience updating unit 133 reads out past experience data in the end-of-episode state from the experience storage unit 122, and recalculates the reward for each piece of read experience data.

[0045] Next, an example of recalculating the reward when the reward function is changed will be described. For example, the experience update unit 133 compares the reward for the episode end state with the reward for each intermediate state and recalculates the reward for each intermediate state based on predetermined conditions. Specifically, the experience update unit 133 uses the reward calculated for the episode end state to compare the reward for the intermediate state transitioned from the start of the episode with the reward calculated for the episode end state, and recalculates the reward for the intermediate state using the maximum value. Furthermore, when the set of experience data is a zero set, the experience update unit 133 initializes the reward for the intermediate state to zero. When experience data for the episode end state is obtained, the experience update unit 133 recalculates the reward for the intermediate state connected to that experience trajectory using the maximum value among the rewards for the episode end states connected to that intermediate state. Note that the experience update unit 133 may use the average, median, or minimum value as the value for recalculating the reward.

[0046] The learning unit 134 updates the parameters of the behavior selection model using the experience data (second experience data) including the reward updated by the experience updating unit 133. For example, the learning unit 134 samples a subset of the experience data from the set of experience data including the reward updated by the experience updating unit 133. Then, the learning unit 134 updates the parameters of the behavior selection model by mini-batch learning using the subset of the experience data.

[0047] Furthermore, for example, the learning unit 134 updates the parameters of the behavior selection model using the experience data (third experience data) stored by the experience storage unit 122. Specifically, the learning unit 134 performs a process similar to the process of updating the parameters of the behavior selection model using the experience data (second experience data) including the reward updated by the experience update unit 133.

[0048] Furthermore, for example, if the number of times the behavior selection model has been learned is less than a threshold after updating the parameters of the behavior selection model, the learning unit 134 performs a search using the behavior selection model until the number of times the behavior selection model has been learned becomes equal to or greater than the threshold, and updates the parameters of the behavior selection model. Note that the learning unit 134 may set an arbitrary number of times as the number of times of learning in advance, or may accept an arbitrary number of times set by the user.

[0049] The output unit 135 outputs the operation status at the end of the episode as a production schedule in the form of a Gantt chart. For example, the output unit 135 outputs products and machines that manufacture the products as an allocation of each manufacturing date as a production schedule. The output unit 135 may also display product IDs and output the operation status at the end of the episode after performing expansion / contraction and shaping of part of the time axis memory.

[0050] [3. Specific examples of processing] First, the reception process performed by the production scheduling device 100 will be described with reference to FIGS.

[0051] The reception unit 131 receives input of setting information, input information, a reward adjustment parameter, and an operation mode from a user. FIGS. 2 and 3 are diagrams illustrating an example of setting information according to an embodiment. For example, as shown in FIG. 2, the reception unit 131 receives "setting information" indicating the configuration of machines and whether machines performing a preceding process and a succeeding process are connected to each other. For example, the setting information associates a preceding process including processes 1 to 4 with a succeeding process including processes 1 to 4. Furthermore, the machines used in each process are set. In the example of FIG. 2, it is set that "machine 1-1, machine 1-2, machine 1-3" are used in the preceding process, process 1, and "machine 1-1, machine 1-2, machine 1-3" are used in the succeeding process, process 1. Furthermore, since "○" is set as the relationship between machine 1-1 of the preceding process, process 1, and machine 2-2 of the succeeding process, process 2, it can be seen that these machines are connected.

[0052] The receiving unit 131 also receives the number of days required for each machine and the processing quantity as setting information from the user, as shown in Fig. 3. The setting information received here includes the number of days required, which indicates the number of days required for each machine to produce a product type in each process, and the processing quantity, which indicates the quantity of products that each machine can produce in one day in each process, as shown in Fig. 3. The example in Fig. 3 indicates that it takes "three days" to produce product type A by machine "1-1" in process "1," and that it takes "one day" to produce product type B by machine "3-2" in process "3." The example in Fig. 3 also indicates that "400" products of product type A are produced per day by machine "1-1" in process "1," and that "500" products of product type B are produced by machine "3-2" in process "3."

[0053] 4 is a diagram showing an example of input information according to the embodiment. For example, as shown in FIG. 4, the receiving unit 131 receives, as input information from a user, an inventory status indicating the inventory quantity of each product at the beginning of the period (day), an operation status indicating the products being produced by each machine in each process at the beginning of the period (day), and order information indicating the order status of the type, quantity, and delivery date of each product associated with an order ID, all linked to an input information ID.

[0054] In the example of Figure 4, the input information ID "0" is linked to and accepted as the inventory status at the beginning of the period: time "10 / 31" and product A "100"; the operation status at the beginning of the period: time "10 / 31", process "1", machine "1-1", product "A"; and the order information: order ID "1", product type "C", quantity "1000", and delivery date "11 / 13".

[0055] FIG. 5 is a diagram showing an example of inventory information according to the embodiment. For example, as shown in FIG. 5, the receiving unit 131 receives inventory information indicating the inventory quantities of products A to C at a time (day). In the example of FIG. 5, the inventory information indicates the inventory quantities of product A "100", product B "600", and product C "250" at time "11 / 1". The state storage unit 123 stores the inventory information and stores changes in the inventory information, such as an increase in the production status of a product according to a state transition for a state output by the behavior selection model, and a decrease in the quantity of a product due to the delivery status of the product.

[0056] Furthermore, the reception unit 131 receives an input of the development stage or the operation stage as the operation mode from the user. Furthermore, when the reception unit 131 receives a selection of the development stage from the user as the operation mode, the reception unit 131 assigns an ID to each of the plurality of pieces of input information from the user and receives the plurality of pieces of input information. On the other hand, when the reception unit 131 receives a selection of the operation stage from the user as the operation mode, the reception unit 131 receives one piece of input information from the user as an input. Here, when the reception unit 131 receives a selection of the development stage or the operation stage as the operation mode, the reception unit 131 may repeatedly receive changes to the reward function from the user to be used for training the behavior selection model.

[0057] A specific example of the remuneration adjustment parameter that the receiving unit 131 receives from the user will be described below with reference to FIG.

[0058] The reception unit 131 receives input of a reward function from the user as a reward adjustment parameter. The experience memory unit 122 stores the state st, action at, state st+1 after the action, and reward rt+1 at time step t as experience data et output from the environment unit 132. Here, the experience memory unit 122 stores the reward function as parameters (reward adjustment parameters) when defining the objectives (evaluation indexes) and constraints (penalty indexes) of production scheduling as a parametric reward function. For example, when an evaluation index list "evaluation index 1, evaluation index 2, ..., evaluation index N" and a penalty index list "penalty index 1, penalty index 2, ..., penalty index M" are stored, the experience memory unit 122 stores them as follows, with weight "i" and weight "j" as the priority of each index and the reward function as a weighted sum. Also, the experience memory unit 122 can set either weight "i" or weight "j" to "0" to not consider the evaluation index or penalty index set to "0". When the reward function is a weighted sum, the reward adjustment parameter becomes the weight.

[0059]

number

[0060] The evaluation index and penalty index in the above formula are feature quantities obtained from the user by deductively analyzing the state st and action at, or the state st+1 after the action, and are reward functions used by the user to judge the quality of the schedule. For example, the receiving unit 131 receives makespan and equipment availability as evaluation index 1 in the evaluation index list, and the number of times delivery deadlines are not met and inventory amount as penalty index 1.

[0061] Furthermore, the behavior selection model storage unit 121 stores a behavior selection model that considers delayed rewards given to a series of states and actions from a time step to the end of the episode, rather than selecting an action that maximizes immediate rewards by providing rewards one-to-one to states and actions during each episode for each time step, and that selects an action that maximizes value. Here, the experience storage unit 122 stores the following equation using a discount rate γ, with rt as the formula for calculating rewards and profits at time t. The experience storage unit 122 also stores a profit calculation formula in which the larger the discount rate, the more rewards are added for future times further away from the current time step. The receiving unit 131 receives a value for the discount rate γ (0 to 1) from the user.

[0062]

number

[0063] Here, the environment unit 132 outputs experience data consisting of a state, an action, a state after the action, and a reward from the action selected by the action selection model. Then, the experience memory unit 122 stores the experience data output by the environment unit 132. Here, the output unit 135 may output to the user the operating state included in the experience data of the episode end state.

[0064] Next, the receiving unit 131 receives the setting information, input information, and adjustment or change of the reward function from the user. Next, the environment unit 132 initializes the operation status and inventory status. The initialization here is not limited to initializing the operation status and inventory status, but may also include setting up a behavior selection model, or setting up past experience data without initializing the operation status and inventory status.

[0065] 6 is a diagram illustrating an example of an experience update process according to an embodiment. When the reward function is changed, the experience update unit 133 recalculates the reward using the changed reward function and past experience data. For example, as shown in FIG. 6, the experience update unit 133 compares the reward for the episode end state with the reward for each intermediate state, and recalculates the reward for each intermediate state based on the maximum reward value of the episode end state as a predetermined condition.

[0066] Here, FIG. 6 illustrates an example in which a schedule is created in which the selection of a machine from October 27th to October 28th is one episode, and then the reward is recalculated. First, in FIG. 6, the machine to be used on October 27th is selected. Here, the reward is calculated at the end of the episode, so it is shown as "0.0." Then, in FIG. 6, the experience is explored, the machine to be used on October 28th is selected, the episode ends, and "0.2" is shown as the reward. Next, as shown in FIG. 6, if the reward function is changed, and the reward at the end of the episode is "0.6," the experience update unit 133 recalculates the reward "0.2" along the experience trajectory by overwriting it with the maximum reward value of the episode end state group, "0.6."

[0067] A specific example in which the output unit 135 outputs a Gantt chart will be described below with reference to FIG.

[0068] FIG. 7 is a diagram showing an example of a Gantt chart according to an embodiment. As shown in FIG. 7, the output unit 135 outputs the allocation of a production schedule as a Gantt chart. For example, the example in FIG. 7 shows the selection of each machine in each process at a time (day) and that each machine will produce one of products A to C. Specifically, the output unit 135 outputs a Gantt chart showing the allocation for time "11 / 1" in which machine 1-1 produces product A in process 1 and machine 1-2 produces product B.

[0069] [4. Flow] Next, the procedure of the production scheduling process by the production scheduling device 100 according to the embodiment will be described using Figures 8 and 9. Note that the behavior selection model in the flowcharts described below will be described as an example using a trained behavior selection model. Figures 8 and 9 are flowcharts showing an example of the production scheduling device 100 according to the embodiment.

[0070] The reception unit 131 of the production scheduling device 100 receives a selection of an operation mode from the user. The reception unit 131 receives a selection of the development stage or the operation stage from the user as the operation mode. Here, Fig. 8 explains the case where the operation stage is selected as the operation mode, and Fig. 9 explains the case where the development stage is selected as the operation mode.

[0071] 8, the reception unit 131 of the production scheduling device 100 receives setting information and input information (step S101), and the behavior selection model storage unit 121 reads out the behavior selection model (step S101A). Next, the environment unit 132 initializes the operation status and inventory status (step S102). Here, initialization is not limited to initializing the operation status and inventory status, but also includes setting up a behavior selection model, and setting up past experience data without initializing the operation status and inventory status.

[0072] Next, the experience update unit 133 determines whether the reward function has been changed (step S103). If the reward function has been changed (step S103: Yes), the experience update unit 133 reads the experience data (step S104). Subsequently, the experience update unit 133 recalculates the reward (step S105), and the environment unit 132 recalculates the reward so as to overwrite the experience data stored in the experience storage unit 122, thereby updating the experience data (step S106).

[0073] On the other hand, if the reward function has not been changed (step S103: No), the experience updating unit 133 executes the process from step S107.

[0074] The experience update unit 133 uses the behavior selection model to select an action (step S107), perform a state transition (step S108), and calculate a reward (step S109). Subsequently, the experience storage unit 122 reads out past experience data (step S110), and the environment unit 132 compares the experience data updated by recalculating the reward in step S105 with the experience data output in steps S107 to S109, and if the experience data output in steps S107 to S109 does not exist in the updated experience data, the experience update unit 133 updates the experience data (step S111) and stores it in the experience storage unit 122 (step S112).

[0075] Note that the environment unit 132 does not update the experience data if the experience data updated in step S105 after recalculating the reward is the same as the experience data output in steps S107 to S109. Subsequently, the learning unit 134 updates the behavior selection model based on the updated experience data (step S113).

[0076] Then, the learning unit 134 determines whether the number of learning times is equal to or greater than a threshold (step S114). If the experience updating unit 133 determines that the number of learning times is less than the threshold, it executes the processing from step S107 again (step S114: No). On the other hand, if the learning unit 134 determines that the number of learning times is equal to or greater than the threshold (step S114: Yes), the updated behavior selection model is stored in the behavior selection model storage unit 121 (step S115). Then, the output unit 135 outputs a Gantt chart showing the schedule (step S116).

[0077] Next, a case where the development stage is selected as the operation mode will be described. When the development stage is selected as the operation mode, after receiving the setting information and input information (step S101), the environment unit 132 may initialize the operating status and inventory status (step S102) without reading out the behavior selection model (step S101A). Note that when the development stage is selected, steps S102 to S116 are the same as those in the operation stage, and therefore will be omitted.

[0078] [5. Effects] The production scheduling device 100 according to the first embodiment is a production scheduling device that creates a production schedule based on a behavior selection model, and includes an experience memory unit 122 that selects actions in the production schedule and stores experience data (first experience data) including the selected actions as tree-structured information that represents state transitions, an experience update unit 133 that recalculates the reward using the reward function and past experience data when the reward function is changed, and a learning unit 134 that updates the parameters of the behavior selection model using experience data (second experience data) including the reward updated by the experience update unit 133.

[0079] As a result, even when the reward adjustment parameter is changed, the production scheduling device 100 can recalculate the reward at the end of the episode for the experience set collected before the change and update the entire experience set based on the recalculated value, so that the experience data before the change can be reused. Furthermore, because the production scheduling device 100 can maintain or reuse the experience data, it can improve the efficiency of searches for learning the reinforcement learning model when the reward adjustment parameter is changed, and can perform efficient scheduling using reinforcement learning.

[0080] The production scheduling device 100 according to the first embodiment further includes an environment unit 132 that selects an action in the production schedule from the pre-update behavior selection model, compares empirical data (third empirical data) containing the selected action with the empirical data (second empirical data), and updates the empirical data (second empirical data) with the empirical data (third empirical data) if the empirical data (third empirical data) is not included, the experience memory unit 122 stores the empirical data (third empirical data) updated by the environment unit 132, and the learning unit 134 updates the parameters of the behavior selection model using the empirical data (third empirical data) stored by the experience memory unit 122.

[0081] As a result, the production scheduling device 100 compares the empirical data obtained by performing a search using the behavior selection model based on the operating status and inventory status received from the user by the receiving unit 131 and the state observed by the environment unit 132, which includes the state of inventory, with the past empirical data, and if the past empirical data does not include the empirical data obtained by performing the search, it updates the past empirical data, thereby enabling learning using empirical data for which the reward has been recalculated. Therefore, scheduling using reinforcement learning can be performed efficiently.

[0082] In the production scheduling device 100 according to the first embodiment, the experience updating unit 133 compares the reward at the end of the episode with the reward for each intermediate state, and recalculates the reward for each intermediate state based on predetermined conditions.

[0083] As a result, the production scheduling device 100 only needs to perform the state analysis processing required for reward calculation on the episode end state, which reduces the calculation cost compared to performing state analysis processing on all intermediate states, and because the reward for the episode end state is reflected in the reward for the episode intermediate state, it can be reflected in the profit for the episode intermediate state regardless of the discount rate. Furthermore, the production scheduling device 100 can update the reward for each intermediate state by recalculating the reward for the episode end state, which allows for efficient scheduling using reinforcement learning.

[0084] (others) [1. Hardware Configuration] The production scheduling device 100 according to the embodiment described above is realized by a computer 1000 configured as shown in FIG. 10, for example. FIG. 10 is a hardware configuration diagram showing an example of a computer that realizes the functions of the production scheduling device 100. The production scheduling device 100 according to the embodiment will be described below as an example. The computer 1000 is connected to an output device 1010 and an input device 1020, and is connected to a calculation device 1030, a cache 1040, a memory 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 via a bus 1090.

[0085] The arithmetic device 1030 operates based on programs stored in the cache 1040 or memory 1050, programs read from the input device 1020, etc., and executes various processes. The cache 1040 is a cache such as a RAM that temporarily stores data used by the arithmetic device 1030 for various calculations. The memory 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is a memory realized by a ROM (Read Only Memory), an HDD (Hard Disk Drive), a flash memory, etc.

[0086] The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a monitor or a printer, and may be realized by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (High Definition Multimedia Interface), etc. On the other hand, the input IF 1070 is an interface for receiving information from various input devices 1020, such as a mouse, keyboard, scanner, etc., and may be realized by a USB, etc.

[0087] For example, the input device 1020 may be realized by a device that reads information from an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory, etc. The input device 1020 may also be realized by an external storage medium such as a USB memory.

[0088] The network IF 1080 has a function of receiving data from other devices via the network N and sending it to the arithmetic device 1030, and also a function of transmitting data generated by the arithmetic device 1030 to other devices via the network N.

[0089] Here, the arithmetic device 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic device 1030 loads a program from the input device 1020 or the memory 1050 onto the cache 1040 and executes the loaded program. For example, when the computer 1000 functions as the production scheduling device 100, the arithmetic device 1030 of the computer 1000 executes the program loaded onto the cache 1040 to realize the functions of the control unit 130.

[0090] Although the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the embodiments of the present application can be embodied in other forms that include the aspects described in the Disclosure of the Invention section and various modifications and improvements based on the knowledge of those skilled in the art.

[0091] [2. Other] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0092] Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, the above-described embodiments can be combined as appropriate within the scope of not causing contradictions in the processing content. [Explanation of symbols]

[0093] 100 Production scheduling device 110 Communications Department 120 Storage section 121 Action Selection Model Memory 122 Experience Memory Department 123 State memory unit 130 Control Unit 131 Reception 132 Environment Department 133 Experience Update Department 134 Learning Department 135 Output section

Claims

1. A production scheduling device that creates a production schedule based on a behavioral selection model, an experience storage unit that selects an action in the production schedule and stores first experience data including the selected action as tree structure information representing state transitions; an experience update unit that, when a reward function is changed, recalculates the reward using the reward function and past experience data; a learning unit that updates parameters of a behavior selection model using second experience data including the reward updated by the experience updating unit; A production scheduling device comprising:

2. an environment unit that selects an action in the production schedule from the action selection model before updating, compares third empirical data including the selected action with the second empirical data, and updates the second empirical data with the third empirical data if the third empirical data is not included; the experience storage unit stores the third experience data updated by the environment unit; The learning unit updates parameters of the behavior selection model using the third experience data stored by the experience storage unit.

2. The production scheduling device according to claim 1.

3. The experience update unit compares the reward at the end of the episode with the reward for each intermediate state, and recalculates the reward for each intermediate state based on a predetermined condition.

2. The production scheduling device according to claim 1.

4. A production scheduling method for formulating a production schedule based on a behavioral selection model, comprising: an experience storage step of selecting an action in the production schedule and storing first experience data including the selected action as tree structure information representing state transitions; an experience update step of recalculating the reward using the reward function and past experience data when the reward function is changed; a learning step of updating parameters of a behavior selection model using second experience data including the reward updated by the experience updating step; A production scheduling method comprising:

5. A production scheduling program that creates a production schedule based on a behavioral selection model, an experience storage step of selecting an action in the production schedule and storing first experience data including the selected action as tree structure information represented by state transitions; an experience update step of recalculating the reward using the reward function and past experience data when the reward function is changed; a learning step of updating parameters of a behavior selection model using second experience data including the reward updated by the experience updating step; A production scheduling program that causes a computer to execute the above.

Images