Planning device, planning method, and program

Abstract

To suppress an increase in time required for creating a plan of conveyance work of an object to be conveyed in a storage space.SOLUTION: The plan creation device 100 selects a conveyance object to be a target of a conveyance work from among conveyance objects in a state of being conveyable by the conveyance means, and derives conveyance information, which is information capable of specifying at least a conveyance order and a conveyance destination of the selected conveyance object, by performing reinforcement learning. The selection of the conveyed object to be the target of the conveyance work includes selection based on a rule determined in advance as a condition of the conveyed object to be the conveyance target.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a planning device, a planning method, and a program. [Background technology]

[0002] A plan for transporting transported objects in a storage yard is often prepared in advance, and the transporting operations are performed based on the plan. For example, in a coil yard, coils are stored stacked in multiple layers to efficiently utilize storage space. A coil is a strip-shaped object (e.g., a metal strip such as a steel strip) wound into a coil. When stacking multiple coils, the coils must be positioned to satisfy various constraints based on the coil's shape, type, packaging, and other factors. Furthermore, the transporting operations of coils in a coil yard include receiving coils into the coil yard, unloading coils from the coil yard, and transporting coils within the coil yard (e.g., transshipping coils). These transporting operations are performed, for example, by one or more cranes. A decrease in the efficiency of the receiving and unloading operations forces a slowdown in the production pace of the preceding and following processes, which can lead to a decrease in productivity. Therefore, smooth coil transport operations, including receiving and unloading operations, are required. To ensure smooth transport, the coil yard must have free space to hold the coils being received. Furthermore, it is desirable that no other coils be placed above the coils being unloaded. However, it is not easy to manually plan efficient transport within limited storage space while satisfying the constraints described above.

[0003] Therefore, there is a need for a technology that enables at least a part of the process of creating a plan for the transport work of the transported object to be performed manually using a computer, etc. Patent Documents 1 and 2 disclose such technology.

[0004] Patent Document 1 discloses that executable work instructions for all buildings are compiled and narrowed down based on the operating status of equipment in a warehouse and work instructions stored in a computer. Specifically, Patent Document 1 discloses that when narrowing down work instructions, if there is a coil above the coil for which a work instruction has been created, a new work instruction is created for the coil above.

[0005] Patent Document 2 discloses that the procedure for moving all of a plurality of containers stored in a yard from the yard to a buffer area is determined using Q-learning, which uses Q values ​​corresponding to the states of the plurality of containers and the operation of moving the plurality of containers from that state to another state. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 7-157018 [Patent Document 2] Patent No. 429109 Summary of the Invention [Problem to be solved by the invention]

[0007] However, with the technology described in Patent Document 1, it is necessary to create executable work instructions at a certain point in time for all buildings in the warehouse, and then check the arrangement of each coil in the warehouse at a later point in time and check each of the work instructions. Therefore, as the scale of the problem for which a plan is to be created increases due to an increase in the number of transported objects, the time required to create the plan increases.

[0008] Furthermore, in the technology described in Patent Document 2, all containers that can be moved to the buffer area are moved to the buffer area, and then work for the remaining containers is determined using Q-learning. Therefore, when the number of containers that can be moved to the buffer area is small, there is a risk that the number of remaining containers will be large. Furthermore, when the scale of the problem for which a plan is to be created becomes large due to, for example, an increase in the number of transported objects, there is a risk that the number of remaining containers will be large. In these cases, it takes a long time to create a plan.

[0009] The present disclosure has been made in consideration of the above-mentioned problems, and aims to prevent an increase in the time required to create a plan for transporting items at a storage yard. [Means for solving the problem]

[0010] The planning device disclosed herein is a planning device that performs processing to create a plan for transportation work that includes at least one of receiving work, which is the work of transporting transported objects to a storage yard, and unloading work, which is the work of transporting transported objects from the storage yard, and is equipped with a selection unit that selects transported objects to be the target of the transportation work from among transported objects that are in a state that can be transported by a transportation means, and a derivation unit that derives transportation information, which is information that can at least identify the transport order and destination of the transported objects selected by the selection unit, by performing reinforcement learning in which a space including the storage yard is used as an environment, the arrangement of the transported objects in the space including the storage yard is used as a state, and the transportation of the transported objects is used as an action, and the selection of the transported objects to be the target of the transportation work by the selection unit includes selection based on rules that are predetermined as conditions for the transported objects to be transported.

[0011] The planning method disclosed herein is a planning method that performs processing to create a plan for transportation work that includes at least one of receiving work, which is the work of transporting transported objects to a storage yard, and unloading work, which is the work of transporting transported objects from the storage yard, and includes a selection step that selects transported objects to be the subject of the transportation work from transported objects that are in a state that can be transported by a transportation means, and a derivation step that derives transportation information, which is information that can at least identify the transportation order and destination of the transported objects selected by the selection step, by performing reinforcement learning in which a space including the storage yard is used as an environment, the arrangement of the transported objects in the space including the storage yard is used as a state, and the transportation of the transported objects is used as an action, and the selection of the transported objects to be the subject of the transportation work by the selection step includes selection based on rules that are predetermined as conditions for the transported objects to be transported.

[0012] The program disclosed herein causes a computer to function as each part of the plan creation device. [Effects of the Invention]

[0013] According to the present disclosure, it is possible to suppress an increase in the time required to create a plan for transporting objects in a storage yard. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of a plan creation device. [Figure 2] FIG. 2 is a diagram showing an example of a coil yard. [Figure 3] 10A and 10B are diagrams illustrating an example of unprocessed coil information and processed coil information. [Figure 4] 10A and 10B are diagrams illustrating an example of storage location attribute information and inventory information. [Figure 5] FIG. 10 is a diagram illustrating an example of a selection target rule. [Figure 6] FIG. 10 is a diagram illustrating an example of an upper storage space constraint. [Figure 7] FIG. 10 is a diagram illustrating an example of an immediate reward. [Figure 8]10 is a flowchart illustrating an example of a plan creation method. [Figure 9A] FIG. 10 is a diagram showing an example of an initial state S0 of the coil yard. [Figure 9B] FIG. 10 is a diagram showing an example of a state S1 of the coil yard. [Figure 9C] FIG. 10 is a diagram showing an example of a state S2 of the coil yard. [Figure 9D] FIG. 10 is a diagram showing an example of a state S3 of the coil yard. [Figure 9E] FIG. 10 is a diagram showing an example of a state S4 of the coil yard. [Figure 9F] FIG. 10 is a diagram showing an example of a state S5 of the coil yard. [Figure 10] FIG. 10 is a diagram showing an example of the initial positions of the storage space, receiving pit, delivery pit, and crane of a coil yard. [Figure 11] FIG. 10 is a diagram showing an example of a coil placed in a storage space in an initial state. [Figure 12] FIG. 10 is a diagram illustrating an example of a scheduled implementation time. [Figure 13] FIG. 10 is a diagram showing an example of a storage space where each coil can be placed. [Figure 14A] FIG. 1 is a diagram illustrating an example of the distance between two locations. [Figure 14B] FIG. 10 is a diagram showing an example of travel time of a crane (coil) between two locations. [Figure 15] FIG. 10 is a diagram illustrating an example of an immediate reward in this specific example. [Figure 16] FIG. 10 is a diagram illustrating an example of processing in the first step (step 1) of the first episode (episode 1). [Figure 17] FIG. 10 is a diagram illustrating an example of processing in the second step (step 2) of the first episode (episode 1). [Figure 18] FIG. 10 is a diagram illustrating an example of the processing of the third step (step 3) of the first episode (episode 1). [Figure 19]FIG. 10 is a diagram illustrating an example of the processing of the fourth step (step 4) of the first episode (episode 1). [Figure 20] FIG. 10 is a diagram showing an example of the processing of the fifth step (step 5) of the first episode (episode 1). [Figure 21] This figure shows an overview of the transportation work performed in each state at the end of the first episode (episode 1), and an example of the immediate reward in each state. [Figure 22] FIG. 10 is a diagram illustrating an example of the processing of the first step (step 1) of the second episode (episode 2). [Figure 23] FIG. 10 is a diagram showing an example of information that can be obtained at the end of the second episode (episode 2). [Figure 24] FIG. 10 is a diagram showing an example of information that can be obtained when the thirteenth episode (episode 13) is completed. [Figure 25] FIG. 10 is a diagram showing an example of conditions for creating an operation plan and a created result. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. Note that the comparison of length, position, size, spacing, etc. being the same includes not only being strictly the same, but also being different within the scope of the gist of this disclosure (for example, being different within the tolerance range determined at the time of design). Also, for convenience of explanation and notation, in each drawing, each configuration may be shown with some of the configuration omitted or at least some of the configuration may be shown with a simplified representation.

[0016] FIG. 1 illustrates an example of the functional configuration of the plan creation device 100. The hardware of the plan creation device 100 includes, for example, one or more hardware processors, such as a central processing unit (CPU), and one or more memories, such as a random access memory (RAM) and a read-only memory (ROM). Various calculations are performed by executing one or more programs stored in the memories using the one or more hardware processors. The hardware of the plan creation device 100 also includes an input device and an output device. The input device acquires various information required for processing in the plan creation device 100. The input device includes, for example, at least one of a user interface, a communication device, and a storage medium. A graphical user interface (GUI) may be used as the user interface. The output device outputs information stored and created in the plan creation device 100 to the outside. The output device includes, for example, at least one of a communication device, a computer display, and a storage medium. The plan creation device 100 may also be implemented using dedicated hardware, such as an application-specific integrated circuit (ASIC).

[0017] The plan creation device 100 is an example of a device that performs processing to create a plan for transportation operations that includes at least one of receiving operations, which are operations for transporting transported objects to a storage yard, and unloading operations, which are operations for transporting transported objects from the storage yard. In the following description, the plan for transportation operations will be referred to as a transportation plan as necessary. The transportation plan may be a plan for receiving operations only, or a plan for unloading operations only, but this embodiment will exemplify a case in which the transportation plan is a plan for transportation operations that includes receiving operations and unloading operations. Furthermore, this embodiment will exemplify a case in which the transportation plan is a plan for transportation operations that further includes, in addition to receiving operations and unloading operations, a reshuffling operation, which is an operation for transporting transported objects within a storage yard.

[0018] Furthermore, the transported object is not limited. The transported object may be, for example, a coil, a slab, a container, or any other object. The storage location for the transported object is a storage location appropriate for the transported object. For example, if the transported object is a coil or a slab, the storage location may be, for example, a yard. Furthermore, if the transported object is a container, the storage location may be, for example, a container storage location at a quay or a storage location (for example, a hold) within the container transport means (for example, a ship). Although the transported object and the storage location for the transported object are not limited in this way, this embodiment illustrates a case where the transported object is a coil and the storage location is a coil yard. Therefore, first, an example of a coil transport operation will be described.

[0019] <Location> Fig. 2 is a diagram showing an example of a coil yard 210. Fig. 2 shows the coil yard as viewed from above. FIG. 2 illustrates an example in which the coil yard 210 is a storage location for coils C to be processed in the processing line 220. For convenience of notation, only one coil C is labeled with the symbol (C) in FIG. 2. FIG. 2 also illustrates an example in which the coil yard 210 includes a receiving area 210a, a discharging area 210b, and a waiting area 210c. FIG. 2 also illustrates an example in which a receiving pit 230a, a receiving / discharging pit 230b, and a discharging pit 230c are located next to the coil yard 210. At least one of the receiving pit 230a, the receiving / discharging pit 230b, and the discharging pit 230c may be located within the coil yard 210.

[0020] A transport means such as a truck transports the coil C from an upstream process on the processing line 220 to the receiving pit 230a or the receiving / delivery pit 230b. FIG. 2 illustrates an example in which the coil C is transported from the receiving pit 230a or the receiving / delivery pit 230b to the receiving area 210a. FIG. 2 illustrates an example in which the coil C transported to the receiving pit 230a or the receiving / delivery pit 230b is placed in the receiving area 210a. In this embodiment, an example in which the transport work of the coil C from the receiving pit 230a or the receiving / delivery pit 230b to the receiving area 210a is the receiving work is illustrated. In the following description, the transport of the coil C from the receiving pit 230a or the receiving / delivery pit 230b to the receiving area 210a will be referred to as receiving or receiving work, as necessary.

[0021] The coil C placed in the receiving area 210a is processed in the processing line 220. The coil C placed in the receiving area 210a may be transported to the waiting area 210c, and then transported from the waiting area 210c to the processing line 220 and processed in the processing line 220. The processing line 220 may include, for example, a process of treating the surface of the coil C, a process of heat-treating the coil C, or both of these processes, or may include other processes. The processing line 220 may also include an uncoiler and a coiler (not shown). In this case, the uncoiler may unwind the coil, the unwound strip-shaped object (e.g., a metal strip such as a steel strip) may be subjected to the above-described processing, and the processed strip-shaped object may be wound into a coil by the coiler. In this way, in this embodiment, an example is shown in which the transported object includes the coil C before being processed in the processing line 220 and the coil C after being processed in the processing line 220. In the following description, the coil C before being processed in the processing line 220 will be referred to as an unprocessed coil C as necessary, and the coil C after being processed in the processing line 220 will be referred to as a processed coil C as necessary. Furthermore, when there is no need to distinguish between the unprocessed coil C and the processed coil C, they may be referred to as coil C as necessary. Note that when the storage location is a slab yard or a container storage location, unprocessed transported materials and processed transported materials are generally not mixed in the storage location.

[0022] FIG. 2 illustrates an example in which the processed coil C is transported from the processing line 220 to the waiting area 210c. The processed coil C may also be transported from the processing line 220 to the unloading area 210b. FIG. 2 also illustrates an example in which, when the processed coil C is transported from the coil yard 210 to a downstream process, the processed coil C is transported from the unloading area 210b to a transport means such as a truck waiting in the receiving / unloading pit 230b or the unloading pit 230c, and then transported to the downstream process by the transport means. That is, FIG. 2 illustrates an example in which the processed coil C is placed in the receiving / unloading pit 230b or the unloading pit 230c. However, for example, if the end time of the unloading operation for the processed coil C is later than the scheduled time of the unloading operation for the processed coil, and the processed coil C is placed in the waiting area 210c, the unloading operation for the processed coil C may be performed. The downstream process may be a process of performing further processing on the processed coil C, or a process of storing the processed coil C in a warehouse or the like. In this embodiment, an example of the dispensing work is shown, in which the work of transporting the processed coil C from the dispensing area 210b or the waiting area 210c to the receiving and dispensing pit 230b or the dispensing pit 230c is taken as an example of the dispensing work. In the following description, the transport of the coil C from the dispensing area 210b to the receiving and dispensing pit 230b or the dispensing pit 230c will be referred to as dispensing or dispensing work, as necessary.

[0023] In this embodiment, among the transport operations in the coil yard 210, excluding the receiving operation, the unloading operation, the transport operation of the unprocessed coil C to the processing line 220, and the transport operation of the processed coil C from the processing line 220, a case will be illustrated in which the transport operation of the coil C necessary to perform the receiving operation and the unloading operation is a rearrangement operation. The rearrangement operation includes, for example, the transport operation of the unprocessed coil C within the receiving area 210a, the transport operation of the unprocessed coil C from the receiving area 210a to the waiting area 210c, the transport operation from the waiting area 210c to the unloading area 210b, the transport operation of the coil C within the waiting area 210c, and the transport operation of the processed coil C within the unloading area 210b.

[0024] 2 illustrates an example in which the coil yard 210 is divided into a receiving area 210a, a discharging area 210b, and a waiting area 210c, but such divisions are not necessary. Also, while FIG. 2 illustrates an example in which the receiving / discharging pit 230b can perform both the receiving operation of unprocessed coils C and the discharging operation of processed coils C, there is no need to have a pit that can perform both the receiving operation and the discharging operation.

[0025] FIG. 2 also illustrates an example in which the coils C are transported in the coil yard 210 using a crane 240, which is an example of a transport means. For convenience of illustration, FIG. 2 only illustrates the hoisting portion of the crane 240. FIG. 2 also illustrates an example in which there is one crane 240. However, there may be a plurality of cranes 240, which are an example of a transport means. In this case, for example, the transport means performing the receiving operation and the transport means performing the unloading operation may be different transport means. Furthermore, the transport means performing a transport operation other than the receiving operation and the unloading operation (for example, the transport operation of the coils C to the waiting area 210c) may be different transport means from at least one of the transport means performing the receiving operation and the transport means performing the unloading operation.

[0026] <Planning device 100> Returning to the description of FIG. 1, an example will be described in which the plan creation device 100 includes an acquisition unit 110, a selection unit 120, a derivation unit 130, and an output unit 140.

[0027] <<Acquisition part 110>> The acquiring unit 110 acquires pre-acquired information, which is information that needs to be acquired in advance so that the plan creation device 100 can perform processing to create a transportation plan. In this embodiment, a case where the acquiring unit 110 acquires the pre-acquired information via an input device included in the plan creation device 100 is illustrated. If the input device includes a communication device, the acquiring unit 110 may acquire the pre-acquired information received by the input device from an external device. If the input device includes a user interface, the acquiring unit 110 may acquire information such as numerical values ​​or characters specified as the pre-acquired information by a user using the user interface. If the input device includes a storage medium, the acquiring unit 110 may acquire the pre-acquired information by reading out the pre-acquired information stored in the storage medium.

[0028] In this embodiment, an example is given of a case where the pre-acquired information includes at least one of information on transported objects to be received and information on transported objects to be sent out, among the transported objects included in the transport plan. Specifically, in this embodiment, an example is given of a case where the pre-acquired information includes information on unprocessed coils C to be received and information on processed coils C to be sent out. In the following description, information on unprocessed coils C to be received will be referred to as unprocessed coil information as necessary, and information on processed coils C to be sent out will be referred to as processed coil information as necessary.

[0029] Fig. 3 is a diagram showing an example of unprocessed coil information and processed coil information. Note that the numerical values ​​shown in Fig. 3 and the subsequent figures are added for convenience, and do not necessarily correspond to the coil yard 210 illustrated in Fig. 2. Furthermore, the numerical values ​​shown in each figure do not necessarily correspond to each other.

[0030] FIG. 3(a) is a diagram showing an example of information included in the unprocessed coil information. In FIG. 3(a), the coil ID is identification information of the unprocessed coil C. In FIG. 3(a), the receiving time is the scheduled time for the receiving work to be performed. The scheduled time for the receiving work may be determined, for example, based on the scheduled arrival time of a transport means such as a truck at the receiving pit 230a or the receiving / unloading pit 230b. Using the scheduled time for the receiving work in this manner is preferable because it allows the timing of the receiving work to be determined in more detail in the transfer plan. However, as long as the order of the receiving work can be specified, it is not necessary to use the scheduled time for the receiving work. For example, the order of the receiving work may be used instead of the scheduled time for the receiving work. In this embodiment, the receiving time is an example of receiving timing information for the unprocessed coil C.

[0031] 3(a), the PIT ID is identification information of the receiving pit 230a or the receiving / delivery pit 230b at which the transportation means transporting the unprocessed coil C is scheduled to arrive. Note that, for example, if there is no receiving pit 230a or receiving / delivery pit 230b, the PIT ID does not need to be included in the unprocessed coil information. In FIG. 3(a), the outer diameter, width, and weight are examples of attribute information of the unprocessed coil C, and are the outer diameter (coil diameter), width (coil width), and weight of the unprocessed coil C, respectively.

[0032] FIG. 3(b) is a diagram showing an example of information included in the processed coil information. In FIG. 3(b), the coil ID is identification information of the processed coil C. In FIG. 3(b), the take-out time is the scheduled time for the take-out operation. By using the scheduled time for the take-out operation in this manner, the timing of the take-out operation can be determined in more detail in the transport plan, which is preferable. However, it is not necessary to use the scheduled time for the take-out operation as long as the order of the take-out operation can be identified. For example, the order of the take-out operation may be used instead of the scheduled time for the take-out operation. In this embodiment, the take-out time is an example of take-out timing information for the processed coil C.

[0033] 3(b), the PIT ID is identification information of the receiving / dispensing pit 230b or the dispensing pit 230c at which the transportation means for transporting the processed coil C is scheduled to arrive. Note that, for example, if one of the receiving / dispensing pit 230b and the dispensing pit 230c does not exist, the PIT ID may not be included in the processed coil information. In FIG. 3(b), the outer diameter, width, and weight are examples of attribute information of the processed coil C, and are the outer diameter (coil diameter), width (coil width), and weight of the processed coil C, respectively.

[0034] Furthermore, in this embodiment, an example is given in which the pre-acquired information includes, as storage yard information, storage yard attribute information and storage yard inventory information at the time of creating a transportation plan. The storage yard inventory information is, for example, information that can identify which coil is placed at which position in the storage yard. Specifically, in this embodiment, an example is given in which the pre-acquired information includes attribute information of the coil yard 210 and inventory information of the coil yard 210 at the time of creating a transportation plan. In the following description, the attribute information of the coil yard 210 will be referred to as storage yard attribute information as needed, and the inventory information of the coil yard 210 at the time of creating a transportation plan will be abbreviated as inventory information as needed.

[0035] FIG. 4 is a diagram showing an example of the storage location attribute information and the inventory information. FIG. 4(a) is a diagram showing an example of information included in the storage location attribute information. In Fig. 4(a), the storage yard ID is identification information of a space in the storage yard where the transported object is placed. In the following description, the space in the storage yard where the transported object is placed will be referred to as a storage yard space as necessary. In this manner, in this embodiment, the coil C is not placed freely in the coil yard 210, but is placed in a storage yard space that is set in advance in the coil yard 210.

[0036] 4(a), the column indicates the identification information of the column of the storage space. The column of the storage space is expressed by the order of the storage spaces in the longitudinal direction (y-axis direction) of the processing line 220, for example, when the storage space closest to the reference position of the coil yard 210 is set as the reference (=1).

[0037] 4(a), the address indicates the identification information of the storage space. The addresses of the storage spaces are expressed as the order of the storage spaces in the width direction (x-axis direction) of the processing line 220, for example, when the storage space closest to the reference position of the coil yard 210 is set as the reference (=1).

[0038] In FIG. 4(a), the row indicates the identification information of the row of each storage space. The rows of storage spaces are expressed, for example, by the vertical arrangement of each storage space with the lowest row being the reference (=1). In this manner, in this embodiment, the space above a coil is also a storage space, and an example is given of a case in which a coil can be placed in that storage space. Note that when a coil is placed on top of a coil, the coil is placed, for example, in a recess formed by arranging two coils side by side. Furthermore, the column and address of the storage space for the coil may be, for example, the column and address of the storage space for the two coils below the coil with the smaller column number. Furthermore, for example, both the column and address of the storage spaces for the two coils below the coil may be included in the column and address of the storage space for the coil.

[0039] In Figure 4(a), the coordinate x indicates the position of each storage space in the x-axis direction. The coordinate x may be, for example, the x-coordinate of the center of each storage space. The x-axis direction is, for example, a direction parallel to the width direction (direction perpendicular to the longitudinal and height directions) of the processing line 220 (see the x-axis direction shown beside the processing line 220 in Figure 2).

[0040] In Figure 4(a), the coordinate y indicates the position of each storage space in the y-axis direction. The coordinate y may be, for example, the y-coordinate of the center of each storage space. The y-axis direction is, for example, a direction parallel to the longitudinal direction of the processing line 220 (see the x-axis direction shown beside the processing line 220 in Figure 2). In FIG. 4(a), outer diameter MIN and outer diameter MAX respectively indicate the minimum and maximum outer diameters of the coil C that can be placed in each storage space.

[0041] FIG. 4(b) is a diagram showing an example of information included in the inventory information. 4(b), as described above, the storage space ID is identification information of the storage space, and the coil ID is identification information of the coil C. In Figure 4(b), the outer diameter, width, and weight are examples of attribute information of coil C placed in the storage area when the transportation plan is created, and are the outer diameter (coil diameter), width (coil width), and weight of coil C, respectively.

[0042] In this embodiment, in addition to the above-described pre-acquired information, the acquiring unit 110 also acquires, as pre-acquired information, information that needs to be set in advance as information to be used in processing in the selecting unit 120 and the derivation unit 130, which will be described later. An example of this information will be described later.

[0043] <<Selection Unit 120>> 1, the selection unit 120 selects an object to be transported from among objects that can be transported by the transport means based on predetermined rules as conditions for the object to be transported. In this manner, in this embodiment, the selection unit 120 selects an object to be transported by using a rule-based method for determining whether or not the object satisfies the conditions for the object to be transported, thereby selecting the object to be transported.

[0044] For simplicity, this embodiment illustrates a case where the crane 240 can transport the coil C on the top tier but cannot transport the coil C on the tier below the top tier. Therefore, this embodiment illustrates a case where the only transportable object that can be transported by the transport means is the coil on the top tier. Note that if no coil is placed on top of a coil, the topmost coil is the coil on the first tier. However, the crane 240 may also be capable of transporting multiple coils C stacked vertically at once. In the following description, a rule that is predefined as a condition for the transported object to be transported is referred to as a selection rule as necessary. The selection rule is included in the pre-acquired information. The selection rule may be determined based on, for example, at least one of the following: the timing of the receiving operation of the transported object to be transported in the receiving operation; the timing of the unloading operation of the transported object to be transported in the unloading operation; the location in the storage area of ​​the transported object to be transported in the unloading operation; and the location in the storage area of ​​the transported object at the time of creating the transport plan.

[0045] FIG. 5 is a diagram illustrating an example of selection target rules. An example of selection target rules will be described with reference to FIG. 5. Note that for convenience of explanation and notation, FIG. 5 shows a simplified configuration of the coil yard 210 (the positional relationship between the delivery area 210b and the waiting area 210c is reversed between FIG. 5 and FIG. 2). In the following description, the processed coil C that is the subject of the delivery operation will be referred to as the coil to be delivered C as needed. The unprocessed coil C that is the subject of the receiving operation will be referred to as the coil to be received C as needed.

[0046] 5 illustrates an example in which the receiving operation is performed in the order of the coils to be received C11, C12, and C13. Also, in FIG. 5, the hatched areas indicate that the coils to be dispensed are processed coils C14, C15, and C16.

[0047] In this embodiment, the following conditions (A) to (G) are included in the selection rule as conditions for the coil C to be transported. (A) The coil C on the upper level of the processed coil C that is the target of the removal operation is defined as the coil C to be transported. 5, coils C17 and C18 are located above coil C15 to be transported, so coils C to be transported include coils C17 and C18.

[0048] (B) Among the multiple coils C to be received, the coil C to be received earliest in the order of the receiving work (in this embodiment, the coil C with the earliest scheduled time for the receiving work) is set as the coil C to be transported. 5, among the multiple coils to be received C11 to C13, the coil C to be received earliest in the order of receiving work is the coil to be received C11. Therefore, the coils C to be transported include the coil to be received C11.

[0049] (C) Of the multiple coils to be dispensed C, the coil to be dispensed C having the earliest order of dispensing operation (in this embodiment, the coil to be dispensed C having the earliest scheduled time of implementation of the dispensing operation) is determined to be the coil C to be transported. 5, for example, if the coil C to be dispensed that is the earliest in the order of dispensing among the coils to be dispensed C14 to C16 is coil C14 or C16, the coils C to be transported include coil C14 or C16. On the other hand, for example, if the coil C to be dispensed that is the earliest in the order of dispensing among the coils to be dispensed C14 to C16 is coil C15, coil C15 is not included in the coils C to be transported. This is because coil C15 is not the coil on the topmost level and is not an object to be transported that can be transported by transport means.

[0050] (D) If there is no storage space available to transport the coil C to be received, whose scheduled time for the receiving work is within a specified time from the scheduled time of the coil C with the earliest order of receiving work, to the receiving area 210a, the unprocessed coil C that needs to be transported to secure the storage space is determined to be the coil C to be transported.

[0051] FIG. 5 illustrates a case where the coils C to be received, whose scheduled time for the receiving work is within a specified time from the scheduled time for the coil C11, which has the earliest order of receiving work, are the coils C11 to C13. FIG. 5 also illustrates a case where, of the storage spaces in the receiving area 210a, the storage spaces in which the coil C to be received can be placed are storage spaces SP1 and SP2 (in FIG. 5, the storage spaces in which no coil C is placed are indicated by dashed lines). FIG. 5 also illustrates a case where, of the coils C11 to C13, the only coil to be received that can be placed in storage space SP1 is the coil C13. FIG. 5 also illustrates a case where, of the coils C11 to C13, the only coils to be received that can be placed in storage space SP2 are the coils C11 to C12.

[0052] In this case, for the coil C13 to be received, storage space SP1 is reserved as a storage space for transporting it to the receiving area 210a. Meanwhile, there is only one storage space (storage space SP2) available for placing the coils C11 to C12 to be received. Therefore, in order to reserve a storage space available for the coils C11 to C12 to be received, it is necessary to transport one of the unprocessed coils C19 to C25, which are placed in the storage space available for the coils C11 to C12, from among the unprocessed coils C on the top row of the receiving area 210a. Therefore, the coils C to be transported include the unprocessed coils C19 to C25.

[0053] (E) The coil C to be dispensed that is outside the dispensing area 210b is set as the coil C to be transported. In the example shown in Fig. 5, among the coils to be dispensed C14 to C16, the coil to be dispensed C that is outside the dispensing area 210b is the coil to be dispensed C16 that is in the waiting area 210c. Therefore, the coils C to be transported include the coil to be dispensed C16. As mentioned above, the coil to be dispensed C15 is not the coil on the top level, and therefore is not an object to be transported that can be transported by the transport means. Therefore, even though the coil to be dispensed C15 is outside the dispensing area 210b, it is not included in the coils C to be transported.

[0054] (F) If the start time of the receiving work in the transportation plan for the coil C to be received is later than the scheduled time of carrying out the receiving work, the coil C to be received is designated as the coil C to be transported. In the example shown in Figure 5, for example, if the start time of the receiving work in the transportation plan for the coil C11 that is scheduled to be received earliest among the multiple coils C11 to C13 is later than the scheduled time of carrying out the receiving work, the coils C to be transported will include the coil C11 that is scheduled to be received.

[0055] (G) If the completion time of the withdrawal work in the transportation plan for the coil C to be withdrawn is later than the scheduled time of carrying out the withdrawal work, the coil C to be withdrawn or the coil C that needs to be transported to carry out the withdrawal work for the coil C to be withdrawn will be the coil C to be transported. 5, for example, if the end time of the dispensing work in the transportation plan for coils to be dispensed C14 and C16 is later than the scheduled implementation time of the dispensing work, the coils C to be transported include coils C14 and C16. Also, for example, if the end time of the dispensing work in the transportation plan for coil C15 is later than the scheduled implementation time of the dispensing work, the coils C to be transported include coils C17 and C18.

[0056] 5, the selection unit 120 selects the coil to be received C11, the coil to be dispensed C16, the coils C17 to C18, and the unprocessed coils C19 to C15 as the coils C to be transported. As mentioned above, if the coil to be dispensed C14 is the coil to be dispensed that has the earliest order of dispensing among the coils to be dispensed C14 to C16, the selection unit 120 also selects the coil to be dispensed C14 as the coil C to be transported.

[0057] The selection unit 120 may select all of the coils C that satisfy the above conditions (A) to (G) as the coils C to be transported. Furthermore, the above conditions (A) to (G) may be prioritized. For example, if there is a coil C that satisfies the conditions (F) and (G), the selection unit 120 may select only the coils C that satisfy the conditions (F) and (G) as the coils C to be transported, even if there is a coil C that satisfies the other conditions.

[0058] 5, the number of coils C to be transported can be reduced compared to when all of the objects that can be transported by the transport means (in this embodiment, all of the coils on the top tier) are the coils C to be transported. The more coils C placed in the coil yard 210, the greater the reduction in the number of coils C to be transported.

[0059] Furthermore, in order to create a transportation plan more efficiently, it is preferable to select candidate destinations for the transported object that will be the subject of the transportation operation. Therefore, in this embodiment, a case will be exemplified in which the selection unit 120 selects candidate destinations for the transported object that will be the subject of the transportation operation based on predetermined rules as conditions for destinations to which the transported object can be transported. In the following description, the predetermined rules as conditions for destinations to which the transported object can be transported will be referred to as destination rules as necessary. Note that there may be one or more candidate destinations for the transported object that will be the subject of the transportation operation.

[0060] In this embodiment, a case where there is a constraint on the storage space where the coil C can be placed is illustrated. In the following description, this constraint is referred to as a destination constraint as necessary. In this embodiment, a case where a condition for satisfying the destination constraint is defined in the destination rule is illustrated.

[0061] As shown in FIG. 4(a), a case is exemplified in which a minimum and a maximum outer diameter of the coil C that can be placed in each storage space are set (see outer diameter MIN and outer diameter MAX shown in FIG. 4(a)). In this manner, this embodiment exemplifies a case in which a destination constraint includes a constraint based on the size of the storage space. In the following description, the constraint based on the size of the storage space will be referred to as a storage size constraint, as necessary.

[0062] Furthermore, in this embodiment, a case is illustrated in which a coil C placed in a storage space on the second level or higher must satisfy constraints based on the attributes of the coil C and the attributes of the coil C immediately below the coil C. In the following description, this constraint will be referred to as an upper level storage constraint as necessary. In this embodiment, a case is illustrated in which an upper level storage constraint is included in the destination constraint.

[0063] FIG. 6 is a diagram illustrating an example of an upper storage space constraint. FIG. 6(a) is a diagram showing an example of an upper storage space constraint related to coil width. FIG. 6(a) illustrates a case where coil C31 with a coil width W3 is placed above coils C32 and C33 with coil widths W1 and W2. In this embodiment, a case where the upper storage space constraint related to coil width is not satisfied when at least one of the following formulas (1) and (2) is satisfied is illustrated. In other words, in this embodiment, a case where the upper storage space constraint related to coil width is satisfied when neither of the following formulas (1) nor (2) is satisfied is illustrated. |W3-W1|>TH1 ···(1) |W3-W2|>TH1 ···(2) TH1 is a positive threshold for the difference in coil width. The upper placement constraint on coil width is a constraint to prevent a coil C with an extremely different coil width from being placed above the coil C below it. The threshold TH1 is set in advance from this perspective.

[0064] FIG. 6(b) is a diagram showing an example of an upper storage space constraint regarding coil weight. FIG. 6(b) illustrates a case where coil C34, weighing M3, is placed on top of coils C35 and C36, weighing M1 and M2, respectively. In this embodiment, a case where the upper storage space constraint regarding coil weight is not satisfied is illustrated when at least one of the following formulas (3) and (4) is satisfied. In other words, in this embodiment, a case where the upper storage space constraint regarding coil weight is satisfied is illustrated when neither of the following formulas (3) and (4) is satisfied. |M3-M1|>TH2 (3) |M3-M2|>TH2 (4) TH2 is a positive threshold for the difference in coil weight. The upper storage constraint on coil weight is a constraint to prevent a coil C whose weight is extremely different from that of the coil C below it from being placed above the coil C below it. The threshold TH2 is set in advance from this perspective.

[0065] FIG. 6(c) is a diagram showing an example of an upper storage space constraint related to coil diameter. FIG. 6(c) illustrates a case where coil C37 with a coil diameter D3 is placed above coils C38 and C39 with coil diameters D1 and D2. In this embodiment, a case where the upper storage space constraint related to coil diameter is not satisfied when at least one of the following formulas (5) and (6) is satisfied is illustrated. In other words, in this embodiment, a case where the upper storage space constraint related to coil diameter is satisfied when neither the following formulas (5) nor (6) is satisfied is illustrated. |D3-D1|>TH3 ···(5) |D3-D2|>TH3 ···(6) TH3 is a positive threshold for the difference in coil diameter. The upper placement constraint on coil diameter is a constraint to prevent a coil C with a coil diameter that is extremely different from the coil C below it from being placed above the coil C below it. The threshold TH3 is set in advance from this perspective. The contents of the transport destination rule and the threshold values ​​TH1 to TH3 are information included in the pre-acquired information.

[0066] 5, coils with relatively large coil widths or diameters and coils with relatively small coil widths or diameters may be placed in the coil yard 210. Threshold values ​​TH1 to TH3 may be set for each of a plurality of categories according to the size of the coils.

[0067] Furthermore, in this embodiment, an example is given of a case where the destination constraint includes that the destination storage space of the coil C to be transported is a storage space other than the storage space above the coil to be dispensing C. In this case, if the destination storage space of the coil C to be transported is a storage space other than the storage space above the coil to be dispensing C, the coil C to be transported satisfies the destination constraint, and if the destination storage space of the coil C to be transported is the storage space above the coil to be dispensing C, the coil C to be transported does not satisfy the destination constraint.

[0068] Furthermore, in this embodiment, a case where the destination constraint includes that only the coil C to be dispensed can be transported to the dispensing area 210b is exemplified. In this case, if the coil C to be transported is a coil C other than the coil C to be dispensed, and the storage space to which the coil C is to be transported is the storage space of the dispensing area 210b, the coil C to be transported does not satisfy the destination constraint.

[0069] 5, the selection unit 120 selects the coil C to be transported from the coil to be received C11, the coil C16 to be dispensed, the coils C17 to C18, and the unprocessed coils C19 to C15. The example shows a case in which the selection unit 120 selects a storage space that satisfies the destination constraint as a candidate storage space (destination) for the coil C to be transported. As mentioned above, if the coil C to be dispensed that has the earliest order of dispensing among the coils C14 to C16 to be dispensed is the coil C to be dispensed, the selection unit 120 also selects the coil C to be dispensed as the coil C to be transported. Furthermore, if there are no candidates for the storage space (destination) for the coil C to be transported, the selection unit 120 may exclude the coil C from the coils C to be transported.

[0070] <<Derivation part 130>> 1, the derivation unit 130 derives transportation information, which is information capable of specifying at least the transportation order and destination of the transported objects selected by the selection unit 120, by performing reinforcement learning in which a space including a storage yard is used as the environment, the arrangement of the transported objects in the space including the storage yard is used as a state (transition state), and transportation of the transported objects is used as an action. Note that the agent may be a transport means or a worker performing the transport work.

[0071] In this embodiment, an example is shown in which the environment in reinforcement learning includes a coil yard 210, a receiving pit 230a, a receiving / delivering pit 230b, and a discharging pit 230c. Furthermore, in this embodiment, an example is shown in which the state in reinforcement learning includes the state (arrangement) of the coil C in the coil yard 210. Furthermore, in this embodiment, an example is shown in which the actions in reinforcement learning include receiving work, discharging work, and redistribution work. The reinforcement learning method is not limited. The reinforcement learning method may be, for example, Q-learning, SARSA, or the Monte Carlo method. Furthermore, the reinforcement learning may be deep reinforcement learning. In this embodiment, an example is shown in which the reinforcement learning method is Q-learning.

[0072] The immediate reward may be determined based on, for example, a guideline for evaluating the quality of a transportation plan. For example, the higher the evaluation value based on the guideline, the larger the value of the immediate reward. The immediate reward may include at least one of a positive reward and a negative reward (loss).

[0073] Furthermore, in this embodiment, the derivation unit 130 derives an immediate reward for the action of transporting the object selected by the selection unit 120 using at least one of a positive reward and a negative reward, values ​​of which are determined for each of a plurality of predetermined categories according to the content of the transport work, and derives transport information, which is information that can at least identify the transport order and destination of the object, based on the derived immediate reward. More specifically, in this embodiment, the derivation unit 130 derives an immediate reward for the action of transporting the coil C selected by the selection unit 120 as the coil C to be transported to the candidate destination (storage space) selected by the selection unit 120, using both a positive reward and a negative reward, values ​​of which are determined for each of a plurality of predetermined categories according to the content of the transport work.

[0074] 7A to 7C are diagrams illustrating an example of an immediate reward. Information showing the contents shown in Fig. 7A to Fig. 7C is included in the pre-acquired information.

[0075] 7(a) is a diagram illustrating an example of a basic remuneration, which is used to calculate a positive remuneration. In Figure 7(a), the immediate reward (positive reward) for the transportation task (action) shown in the task content column is a positive reward (r i ) column.

[0076] FIG. 7(b) is a diagram illustrating an example of the transportation cost. The transportation cost is used to calculate the negative reward. In FIG. 7(b), the values ​​shown in the description column and the cost weight (w i ) column and , the immediate reward (negative reward) for the content of the transportation work shown in the corresponding description column is determined by multiplying the value shown in the corresponding description column.

[0077] In Figure 7(b), the x-axis movement distance and the y-axis movement distance are the movement distances of the transport means (crane 240) in the x-axis and y-axis directions, respectively. Furthermore, "no load" indicates that the transport means (crane 240) moves without a load (i.e., without a load). Furthermore, "with load" indicates that the transport means (crane 240) moves with a load (i.e., with a load). In FIG. 2, when the crane 240 at position P1 moves to the receiving / discharging pit 230b (position P2) without a load and transports the coil C to be received from the receiving / discharging pit 230b to the storage space at position P3, the x-axis movement distance (no load) is the distance in the x-axis direction from position P1 to position P4, the x-axis movement distance (with load) is the distance in the x-axis direction from position P2 to position P5, the y-axis movement distance (no load) is the distance in the y-axis direction from position P4 to position P2, and the y-axis movement distance (with load) is the distance in the x-axis direction from position P5 to position P3. Note that position P4 is a position whose y-coordinate is the same as position P2 and whose x-coordinate is the same as position P1. Position P5 is a position whose y-coordinate is the same as position P3 and whose x-coordinate is the same as position P2. Furthermore, position P2 is, for example, the center position of the receiving / discharging pit 230b. Position P3 is, for example, the center of the storage space. Note that the greater the workload of the transport work, the greater the x-axis movement distance and the y-axis movement distance.

[0078] 7(b), the receiving / delivering delay time indicates the delay time of the receiving operation start time in the transportation plan for the coil C to be received relative to the scheduled time of the receiving operation, and the delay time of the unloading operation end time in the transportation plan for the coil C to be unloaded relative to the scheduled time of the unloading operation. Note that the greater the degree of delay from the scheduled time of the receiving operation and unloading operation, the greater the value of the receiving / delivering delay time.

[0079] Furthermore, in FIG. 7(b), "receiving work and reassignment work are not possible" indicates that there is no storage space to which the coil C to be received can be transported, and that there is no storage space to which the coil C to be reassigned can be transported. In this case, the content of the transport work is whether the transport work is one for which receiving work is not possible and whether the transport work is one for which reassignment work is not possible. Note that "receiving work and reassignment work are not possible" corresponds to "the transport work is not feasible," and "receiving work and reassignment work are possible" corresponds to "the transport work is feasible." Since "receiving work and reassignment work are not possible" is realized by an immediate reward (negative reward), it is preferable to set a very large negative reward as the immediate reward.

[0080] 7(c) is a diagram showing an example of an additional reward. In this embodiment, a case where the additional reward is a positive reward is illustrated. FIG. 7(c) illustrates a case where an immediate reward (positive reward) for the content of the transportation work shown in the description column is determined by using the value shown in the description column.

[0081] In Fig. 7(c), the distance (x-axis direction) from the delivery pit of the storage space of the transport source is d x1 is the distance in the x-axis direction from the location of the storage space from which the coil C to be dispensed is transported to the location of the pit to which the coil C to be dispensed is transported, either the receiving / dispensing pit 230b or the dispensing pit 230c. The distance (x-axis) d from the dispensing pit to the storage space to which the coil C is transported x2 is the distance in the x-axis direction from the location of the storage space to which the coil C to be dispensed is to be transported to the location of the pit to which the coil C to be dispensed is to be transported, either the receiving / dispensing pit 230b or the dispensing pit 230c. x1 and d x2 The absolute value of the difference between x1 -d x2 The smaller |) is, the more the coil C to be dispensed is moving in the x-axis direction so as to be closer to the pit (dispensing position) to which the coil C to be dispensed is transported, either the receiving and dispensing pit 230b or the dispensing pit 230c.

[0082] In addition, in Fig. 7(c), the distance (y-axis direction) from the delivery pit of the storage space of the transport source is d y1 is the distance in the y-axis direction from the location of the storage space from which the coil C to be dispensed is transported to the location of the pit to which the coil C to be dispensed is transported, either the receiving / dispensing pit 230b or the dispensing pit 230c. The distance (y-axis) d from the dispensing pit to the storage space to which the coil C is transported y2 is the distance in the y-axis direction from the location of the storage space to which the coil C to be dispensed is to be transported to the location of the pit to which the coil C to be dispensed is to be transported, either the receiving / dispensing pit 230b or the dispensing pit 230c. y1 and d y2 The absolute value of the difference between y1 -d y2 The smaller |) is, the more the coil C to be dispensed is moving in the y-axis direction so as to be closer to the pit (dispensing position) to which the coil C to be dispensed is transported, either the receiving and dispensing pit 230b or the dispensing pit 230c.

[0083] In this embodiment, a case will be illustrated in which the immediate reward for the action of transporting the coil C is derived using the following equations (7) to (10).

[0084]

number

[0085] In equation (7) (and equation (8)), reward base is the basic remuneration. In equation (8), b i and r i i in Fig. 7(a) represents the task division (i) and the positive reward (r i ) corresponds to i in b i is a 0-1 variable that is 1 when the content of the transport work of the coil C is the work content corresponding to the value of i shown in the work category (i) column of FIG. 7(a), and is 0 in other cases. For example, when the content of the transport work of the coil C is the work of receiving it into the receiving area 210a, b1 is 1. i is the positive reward (r i) column. The right side of equation (8) is i ·b i In the example shown in FIG. 7(a), i is an integer between 1 and 6.

[0086] In equation (7) (and equation (9)), -cost is the transportation cost. In equation (9), w j and c j j is the cost category (j) and cost weight (w j ) corresponds to j. j are c1 to c7 shown in the item column of Figure 7(b), and are the values ​​shown in the explanation column. For example, c1 is the value of the x-axis movement distance (without a suspended load). w j is the cost weight (w i ) column value. In equation (7) (and equation (10)), +reward add is an additional reward.

[0087] In this embodiment, in order to derive the transportation cost illustrated in FIG. 7(c) and the additional reward illustrated in FIG. 7(d), the derivation unit 130 performs a transportation simulation (calculation simulating the operation of the transportation means) of the behavior of transporting the transported object selected by the selection unit 120. In the transportation simulation, for example, a determination may be made as to whether the transported object can be transported based on the upper storage space constraints illustrated in FIG. 6, for example. In this case, for example, a transport route for the transported object that can be transported is determined. Then, at least one of the transport distance and the operation time when the transported object is transported along the transport route may be derived. Furthermore, for example, at least one of the work start time and the work end time for each transported object may be derived based on a reference time (e.g., the scheduled time for the receiving operation and the unloading operation) and the operation time for each transported object.

[0088] As described above, in this embodiment, the case where the reinforcement learning method is Q-learning is exemplified. When an action a is taken in state S, an immediate reward r (reward in equation (7)) is obtained and state S transitions to state S', and the Q value (value of the action-value function) Q[S, a] is updated by the following equation (11).

[0089]

number

[0090] In (11), α is the learning rate, and γ is the time discount rate. The Q-learning method itself is realized by a known method (Equation (11) itself is a known equation). As the Q-learning method, for example, the ε-greedy method may be used. In this case, the Q-value is randomly selected with a probability based on the random selection rate ε. In other cases, the maximum Q-value in each state of the Q-table is selected. The learning rate α, the time discount rate γ, and the random selection rate ε are included in the pre-acquired information.

[0091] In this embodiment, an example is given of a case where, in reinforcement learning, when the Q value (value of the action value function) Q[S, a] associated with the state S (placement) and the action a (transportation work) for the transported object in the state S has not been derived, the selection unit 120 selects the transported object to be the target of the transporting work from among the transported objects in the state S that can be transported by the transport means, based on rules that are predetermined as conditions for the transported object to be transported.

[0092] On the other hand, this embodiment illustrates a case where, in reinforcement learning, when a Q value (value of an action-value function) Q[S, a] associated with a state S (disposition) and an action a (transportation task) on a transport object in the state S is derived, a transport object to be a target of the transport task is selected from the actions a. That is, this embodiment illustrates a case where, when a Q value (value of an action-value function) Q[S, a] associated with an action a on a transport object in the state S is derived, a transport object to be a target of the transport task is selected by a method defined in a reinforcement learning algorithm without using the above-mentioned rules.

[0093] In this embodiment, when a transport operation for a coil C in a certain arrangement has not been derived, the selection unit 120 selects, from among the coils C in the arrangement, a coil C that satisfies the conditions (A) to (G) described above as a coil C to be transported, and selects a candidate storage space that satisfies the destination constraints described with reference to FIG. 6 as a candidate destination for the coil C. In this case, the selection unit 120 makes such a selection when the derivation unit 130 processes the first episode in reinforcement learning. Note that such a selection may also be made when the derivation unit 130 processes the second or subsequent episode in reinforcement learning.

[0094] In addition, this embodiment illustrates a case where the derivation unit 130 derives the start time and end time of each transport task based on the scheduled times for the receiving and unloading tasks and the task duration of each transport task. The task duration of a transport task includes, for example, the travel time of the transport means (crane 240 in this embodiment). When deriving the task duration of a transport task, the time required to attach the transported object to the transport means and the time required to remove the transported object from the transport means may be taken into consideration. Furthermore, if the transport means is a crane, the time required to lower (hoist) the transported object and the time required to hoist (lower) the transported object may be taken into consideration when deriving the task duration of a transport task. These times are derived, for example, by a transport simulation. These times may also be set in advance using past performance data, etc.

[0095] The derivation unit 130 derives, by a reinforcement learning algorithm, the contents of all transportation operations (at least the transportation order and the transportation destination) to be included in the transportation plan, among the receiving operations and the unloading operations, from the set of actions for transporting the coil C to be transported selected by the selection unit 120 as described above to the destination (storage space) of the coil C selected by the selection unit 120. As described above, this embodiment illustrates a case in which Q-learning is used as the reinforcement learning method. Furthermore, this embodiment illustrates a case in which the implementation time of the transportation operations (the start time and the end time of the operations) is included in the contents of the transportation operations. Furthermore, this embodiment illustrates a case in which information capable of identifying the implementation time (the start time and the end time of the operations) and the destination of the transportation operations of the coil C to be received, the coil C to be unloaded, and the coil C that needs to be transported in order to perform the receiving operation and the transportation operation (the coil C to be rearranged) is derived as an example of transportation information.

[0096] <<Output Unit 140>> Returning to the explanation of FIG. 1 , the output unit 140 outputs the transportation information derived by the derivation unit 130. In this embodiment, an example is shown in which the output unit 140 acquires the transportation information to an output device included in the plan creation device 100. If the output device includes a computer display, the output device may display the transportation information. If the output device includes a communication device, the output device may transmit the transportation information to an external device. If the output device includes a storage medium, the output device may store the transportation information.

[0097] <Planning method> Next, an example of a plan creation method performed by using the plan creation device 100 of this embodiment will be described with reference to the flowchart in Fig. 8. As described above, this embodiment illustrates a case where the reinforcement learning method is Q-learning. Also, Fig. 8 illustrates a case where the ε-greedy method is used as the Q-learning method. As described above, the reinforcement learning method is not limited to Q-learning.

[0098] In step S801, the acquisition unit 110 acquires the pre-acquired information. Next, in step S802, the derivation unit 130 specifies that the first episode (episode 1) be processed. Next, in step S803, it is specified to perform processing of the first step of the episode. Note that when a step indicates processing of each block in FIG. 8, an S is added after the step (e.g., step S801). On the other hand, when a step indicates a step in reinforcement learning, an S is not added after the step.

[0099] Next, in step S804, the selection unit 120 determines whether or not the Q value (value of the action-value function) Q[S, a] for the state S and action a∈A corresponding to the current step has been registered in the Q table. Here, set A is a set of action candidates in state S derived by a rule-based method. Note that if the state S corresponding to the current step has been registered, the associated action set does not change, so in step S804, it may be determined whether or not the state S corresponding to the current step has been registered in the Q table.

[0100] If the result of the determination in step S804 is that the Q-value Q[S, a] for the state S and action a∈A corresponding to the current step has already been registered in the Q-table (YES in step S804), the processes in steps S805 to S806 are omitted, and the process in step S807, which will be described later, is performed. On the other hand, if the Q-value Q[s, a] for the state S and action a corresponding to the current step has not already been registered in the Q-table (NO in step S804), the process in step S805 is performed.

[0101] In step S805, the selection unit 120 selects a candidate task by using a rule-based method. The selection of a candidate task includes selecting an object to be transported from among objects that can be transported by the transport means. In addition, this embodiment illustrates a case where the selection of a candidate task includes selecting a candidate destination for the object to be transported. More specifically, this embodiment illustrates a case where the selection unit 120 selects a coil C that satisfies the conditions (A) to (G) described above as the coil C to be transported, and selects a candidate storage space that satisfies the destination constraints described with reference to FIG. 6 as a candidate destination for the coil C. In this case, the candidate task includes one or more coils C to be transported and one or more candidate destinations for the coil C.

[0102] Next, in step S806, the derivation unit 130 derives an immediate reward r for the transport task (action) according to the candidate task selected in step S805, and derives a Q-value Q[s, a] using the immediate reward r according to equation (11) and registers it in the Q table. In this embodiment, a case where the immediate reward r (reward) is derived according to equations (7) to (10) is illustrated. Note that when step S806 is performed, the Q-value Q[s, a] is not registered in the Q table, so (1-α)Q[s, a] and Q[s', a'] in equation (11) are 0. In addition, the Q-value registered in the Q table in step S806 is, for example, a value obtained by multiplying the learning rate α by the immediate reward r for the transport task of coil C (coil C for which transport task is essential for receiving and dispensing) that satisfies the conditions (A) to (G) described above.

[0103] Next, in step S807, the derivation unit 130 determines whether or not to randomly select action a in state S corresponding to the current step, using the random selection rate ε. If the result of this determination is that action a is not to be randomly selected (NO in step S807), the process of step S808 is performed. In step S808, the selection unit 120 selects from the Q table an action (transportation work) corresponding to the maximum Q value among the Q values ​​of state S corresponding to the current step. Then, the process of step S810, which will be described later, is performed.

[0104] On the other hand, if the result of the determination in step S807 is that action a is to be selected randomly, the process proceeds to step S809. In step S809, the selection unit 120 randomly selects action a in the state S corresponding to the current step based on the random selection rate ε. Then, the process proceeds to step S810.

[0105] In step S810, the derivation unit 130 determines whether the Q-value Q[S', a'] for the next state S' of the state S corresponding to the current step and the action a'∈A' has already been registered in the Q-table. Here, set A' is a set of task candidates in state S' derived by a rule-based method. If the result of this determination is that the Q-value Q[S', a'] for the next state S' of the state S corresponding to the current step and the action a' has not already been registered in the Q-table (NO in step S810), the process of step S811 is omitted and the process of step S812, which will be described later, is performed.

[0106] On the other hand, if the result of the judgment in step S810 is that the Q value Q[S', a'] for the next state S' and action a' of the state S corresponding to the current step has already been registered in the Q table (YES in step S810), processing in step S811 is performed.

[0107] In step S811, the derivation unit 130 derives the immediate reward r for the action a (transportation work) selected in step S808 or S809 in the Q table, and updates the Q table by updating the Q value Q[S, a] stored in the Q table according to equation (11). Note that this embodiment illustrates a case where the immediate reward r (reward) is derived according to equations (7) to (10).

[0108] Next, in step S812, the derivation unit 130 updates the state (step). In this embodiment, updating the state corresponds to updating the arrangement of the coil C in the coil yard 210 when the transport work corresponding to the action a selected in step S808 or S809 is performed. In addition, the derivation unit 130 derives the work start time and the work end time of the transport work corresponding to the action a selected in step S808 or S809.

[0109] Next, in step S813, the derivation unit 130 determines whether all of the receiving and unloading operations included in the transfer plan have been completed. In the present embodiment, a case is exemplified in which the derivation unit 130 determines whether all of the receiving operations for the coils C to be received and all of the unloading operations for the coils C to be unloaded have been completed in the transfer plan. For example, the derivation unit 130 determines whether the processing of the current episode has been completed. As a result of this determination, if all of the receiving and unloading operations included in the transfer plan have not been completed (NO in step S813), the processing of step S804 described above is performed again. In this case, the processing of steps S805 to S811 is performed based on the state (step) updated in step S812, and the state (step) is updated in step S812.

[0110] On the other hand, as a result of the determination in step S813, if all of the transportation operations included in the transportation plan among the receiving operations and unloading operations have been completed in the transportation plan (YES in step S813), the processing of step S814 is performed. In step S814, the derivation unit 130 determines whether or not to terminate learning. The determination condition for whether or not to terminate learning may be a condition determined in a reinforcement learning (Q-learning in this embodiment) algorithm. For example, the determination condition for whether or not to terminate learning may be a condition that the learning is terminated when the Q table at the end of the previous episode and the Q table at the end of the current episode are the same. Furthermore, for example, the determination condition for whether or not to terminate learning may be a condition that the number of episode repetitions is a predetermined number.

[0111] As a result of step S814, if the learning is not to be ended (NO in step S814), the process of step S815 is performed. In step S815, the derivation unit 130 updates the episode. At this time, the derivation unit 130 returns the state to the initial state. In this embodiment, the arrangement of the coils C in the coil yard 210 is returned to the initial state. Next, in step S816, the derivation unit 130 updates the random selection rate ε. When the process of step S816 ends, the process of step S803 described above is performed. Then, the processes of steps S804 to S816 are performed for the next episode.

[0112] On the other hand, as a result of step S814, if learning is to be ended (YES in step S814), the process of step S817 is performed. In step S817, the output unit 140 outputs the transportation information derived by the derivation unit 130. When the process of step S817 ends, the process according to the flowchart of FIG. 8 ends.

[0113] <Example> Next, a specific example of the plan creation method in this embodiment will be described. In this specific example, as an example of reinforcement learning, a case where Q-learning is performed under the conditions of learning rate α = 0.50, time discount rate γ = 0.99, and random selection rate ε = 0 will be illustrated. In this case, the processes of S807, S809, and S816 in FIG. 8 are unnecessary.

[0114] <<Premise>> First, the premise of the planning method of this specific example will be described. The contents described in this section correspond to the contents of the advance acquisition information acquired in step S801, for example. 9A to 9F are diagrams showing an example of a transition of a state (the arrangement of the coil C in the coil yard 210). The state transition order is Fig. 9A, Fig. 9B, Fig. 9C, Fig. 9D, Fig. 9E, and Fig. 9F. As with Fig. 5, Figs. 9A to 9F also show a simplified configuration of the coil yard 210. Furthermore, the circled numbers (1 to 11) in Figs. 9A to 9F indicate identification information of the storage spaces. In this specific example, for ease of explanation, only 11 storage spaces of the storage spaces in the coil yard 210 will be used as storage spaces in the following explanation. The 11 storage spaces will be referred to as storage spaces 1 to 11 as necessary. The numbers 1 to 11 in storage spaces 1 to 11 correspond to the circled numbers in Figs. 9A to 9F. In this example, for ease of explanation, it is assumed that there is one receiving pit 230a and one dispensing pit 230c, but no receiving / dispensing pit 230b. Note that the numerical values ​​in each figure showing this example, such as Figure 10, are given for the sake of convenience of explanation and do not necessarily match the arrangements in Figures 9A to 9F.

[0115] FIG. 9A is a diagram showing an example of the initial state S0 of the coil yard 210. Note that the pre-acquired information is information identified from FIG. 9A, and information identified from FIGS. 9B to 9F is not included in the pre-acquired information. FIG. 10 is a diagram showing an example of the positions (x-coordinates and y-coordinates) of storage spaces 1 to 11, the receiving pit 230a, the discharging pit 230c, and the crane 240 of the coil yard 210 in the initial state S0. In FIG. 10, 1 to 11 in the location column indicate storage spaces 1 to 11. Furthermore, in, out, and cr_ini in the location column indicate the receiving pit 230a, the discharging pit 230c, and the crane 240, respectively. FIG. 11 is a diagram showing an example of coils C placed in storage spaces 1 to 11 in the initial state S0. In FIG. 10, the symbols (b1, e1, b2, e2) shown in the coil column are identification information for coil C (the symbols shown in the coil column in other figures are also identification information for coil C). In FIGS. 9A to 9F, these symbols are shown as the symbols for coil C. Therefore, in the following description, coil C will be referred to as coil e1, etc., as necessary.

[0116] FIG. 12 is a diagram showing an example of scheduled execution times. As shown in FIG. 12(a), this specific example illustrates a case where the start time of the transport work in the transport plan is 7:00. Also, as shown in FIG. 12(b), this specific example illustrates a case where the coil C to be received is coil i1, and the scheduled time of the receiving work (scheduled receiving time) for the coil i1 to be received is 7:50. Also, as shown in FIG. 12(c), this specific example illustrates a case where the coils C to be dispensed are coils e1 and e2, and the scheduled times of the dispense work (scheduled dispense times) for the coils e1 and e2 to be dispensed are 7:55 and 8:00, respectively. Note that, as shown in FIGS. 9A and 11, in the initial state S0, the coils e1 and e2 to be dispensed are placed in storage spaces 7 and 10, respectively.

[0117] Fig. 13 is a diagram showing an example of the storage spaces in which coils b1, b2, e1, e2, and i1 can be placed. In Fig. 13, a circle (◯) is marked in the column of the storage space in which coil C can be placed, and a cross (×) is marked in the column of the storage space in which coil C cannot be placed. For example, coil b1 can be placed in storage spaces 5 to 11, but cannot be placed in storage spaces 1 to 4.

[0118] FIG. 14A is a diagram showing an example of the distance between two locations. FIG. 14B is a diagram showing an example of the travel time (seconds) of the crane 240 (coil C) between two locations. In FIGS. 14A and 14B, "crane initial" indicates the position (initial position) of the crane 240 in the initial state S0. In this specific example, for ease of explanation, a case is illustrated in which the transport time between two locations is the work time required for transporting the coil C from one of the two locations to the other.

[0119] In this specific example, an example is shown in which an immediate reward for the action of transporting the coil C is derived from the above-mentioned formulas (7) to (10). However, the following points are different from the example shown in FIG. 7. FIG. 15 is a diagram illustrating an example of the immediate reward in this specific example. FIG. 15(a) is a diagram illustrating an example of the basic reward. In contrast to FIG. 7(a), FIG. 15(a) shows a case in which a positive reward (r i ) column is different. Figure 15(b) is a diagram illustrating an example of transportation costs. Items c2, c3, and c4 in Figure 15(b) are the same as items c5, c6, and c7 in Figure 7(b), respectively. In Figure 15(b), the crane travel distance is the sum of the travel distances of the transportation means (crane 240) in the x-axis and y-axis directions. In this way, Figure 15(b) illustrates an example in which the transportation cost -cost((w1·c1)) is derived without distinguishing between a state in which the crane 240 is suspending a load and a state in which the crane 240 is not suspending a load.

[0120] Furthermore, in this specific example, in the processing of the first episode (episode 1), a coil C that satisfies the conditions (A) to (G) described above is selected as the coil C to be transported, and a candidate storage space that satisfies the destination constraints described with reference to Fig. 6 is selected as a candidate destination for the coil C. However, if there is a coil C that satisfies the conditions (F) and (G) described above, even if there is a coil C that satisfies other conditions, only the coil C that satisfies the conditions (F) and (G) is selected as the coil C to be transported. In addition, this specific example illustrates a case where the Q value registered in the Q table during the processing of the first episode (episode 1) is the Q value for the transport work of coil C (coil C for which transport work is essential for receiving and dispensing) that satisfies the conditions shown above (A) to (C), and the Q value for the transport work for which the immediate reward r is a positive value.

[0121] <<Solution>> In this specific example, a case where a solution is found (a transportation plan is created) under the above assumptions will be illustrated. << <episode1>>> 16 is a diagram illustrating an example of the processing of the first step (step 1) of the first episode (episode 1). When processing of step 1 of episode 1 starts, nothing is registered in the Q table. FIG. 16(a) is a diagram showing an example of coil C to be transported selected by the selection unit 120 in step 1 of episode 1. Here, coils i1, b1, and b2 that satisfy conditions (A) and (B) among the conditions (A) to (G) described above are selected. Note that coils b1 and b2 are located above coils e1 and e2 to be transported, so coils e1 and e2 are not transported objects (coils) that can be transported by the transport means. Therefore, even if they satisfy the condition (C) described above, coils e1 and e2 to be transported are not selected as coil C to be transported. Furthermore, there are six candidate storage spaces: storage spaces 2, 3, 4, 6, 8, and 11 (see FIGS. 9A and 13). FIG. 16(a) is an example of information obtained by the processing of step S805, for example.

[0122] FIG. 16(b) is a diagram showing an example of the immediate reward r(reward) derived by the derivation unit 130 in step 1 of episode 1. For example, the immediate reward for the action of transporting the to-be-received coil i1 to the storage space 2 is derived as follows. First, since the transport work in this case is a receiving work into the receiving area 210a, from FIG. 15(a), the basic reward base is 20000.

[0123] The transportation cost -cost for the receiving work of the coil i1 to be received is the sum of the travel distance from the initial position of the crane 240 to the receiving pit 230a and the travel distance from the receiving pit 230a to the storage space 2, and the cost weight w j The value is the product of and multiplied by -1.

[0124] From Figure 15(b), the cost weight w j is 1. Also, from FIG. 14A, the travel distance from the initial position of the crane 240 to the receiving pit 230a is 1000, and the travel distance from the receiving pit 230a to the storage space 2 is 700. Therefore, the transportation cost -cost for the receiving work of the coil i1 to be received is -1700 (=-1×1×(1000+700)).

[0125] In addition, the additional reward for the receiving work of the scheduled receiving coil i1 is 0. From the above, the immediate reward for the action of transporting the to-be-received coil i1 from the receiving area 210a to the storage space 2 is 18300 (=20000-1700+0) according to equation (7). FIG. 16(b) is an example of information obtained by the processing in step S805.

[0126] FIG. 16(c) is a diagram showing an example of a Q table derived by the derivation unit 130 in step 1 of episode 1. In FIG. 16(b), coil C that satisfies the conditions (A) to (C) described above is coil i1. Of the transport tasks other than the transport task for coil C that satisfies the conditions (A) to (C) described above, the transport task for which the immediate reward r is a positive value is the transport task (rearrangement task) of coil b1 to the storage space 11. Therefore, as shown in FIG. 16(c), the Q value for the transport task (receiving task) of coil i1 and the Q value for the transport task (rearrangement task) of coil b1 to the storage space 11 are registered in the Q table. FIG. 16(c) is an example of information obtained by the processing in step S806.

[0127] Fig. 16(d) is a diagram showing an example of the state update result in step 1 of episode 1. Among the Q values ​​of the initial state S0, which is the state corresponding to the current step (step 1), the maximum Q value is 9150. The transport operation corresponding to this Q value is the receiving operation of transporting the coil i1 to be received from the receiving pit 230a to the storage space 2. Therefore, the initial state S0 shown in Fig. 11 is updated to the state S1 illustrated in Fig. 16(d). FIG. 16(d) shows an example of information obtained by the process of step S812 after the determination of NO in step S810.

[0128] Fig. 16(e) is a diagram showing an example of the transport operation at the time when step 1 of episode 1 is completed (the time when the state transitions from the initial state S0 to state S1). In Fig. 16(e), "from" indicates the source storage space or receiving pit 230a, and "to" indicates the destination storage space or dispensing pit 230c (this notation is the same in other figures). Fig. 9B is a diagram showing an example of state S1 of the coil yard 210.

[0129] Here, since the scheduled execution time (7:50) is set for the receiving work (see FIG. 12(b)), the derivation unit 130 derives this scheduled execution time as the work start time. Furthermore, from FIG. 14B, the derivation unit 130 derives the work end time as the time (7:51:56) obtained by adding the scheduled execution time (7:50) plus 63 seconds, which is the time it takes for the crane 240 to move from its initial position to the receiving pit 230a, and 53 seconds, which is the time it takes for the crane 240 to move from the receiving pit 230a to the storage space 2. Note that the work start time of the receiving work is the scheduled execution time of the receiving work, so the work start time of the receiving work is not delayed relative to the scheduled execution time of the receiving work, and therefore the work delay time is 0. FIG. 16(e) shows an example of information obtained by the process of step S812 after the determination of NO in step S810.

[0130] 17 is a diagram illustrating an example of the processing of the second step (step 2) of the first episode (episode 1). When the processing of step 2 of episode 1 starts, the state S1 corresponding to step 2 and the action a for the state S1 are not registered in the Q table. FIG. 17(a) is a diagram showing an example of the coil C to be transported selected by the selection unit 120 in step 2 of episode 1. Here, of the conditions (A) to (G) described above, coils b1 and b2 that satisfy condition (A) are selected. Furthermore, there are two candidate storage spaces: storage spaces 8 and 11 (see FIG. 9B and FIG. 13). FIG. 17(a) is an example of information obtained by processing step S805 after the determinations of NO in steps S813 and S804, respectively. Note that storage space 6 is not included in the candidate storage spaces in FIG. 17(a) because the coil e1 to be dispensed is located below storage space 6, and therefore storage space 6 does not satisfy the destination constraint.

[0131] 17(b) is a diagram showing an example of the immediate reward r (reward) derived by the derivation unit 130 in step 2 of episode 1. FIG. 17(b) is an example of information obtained by the process of step S805, for example. Fig. 17(c) is a diagram showing an example of the Q table derived by the derivation unit 130 in step 2 of episode 1. Fig. 17(c) is an example of information obtained by the processing of step S806, for example.

[0132] FIG. 17(d) is a diagram showing an example of the state update result in step 2 of episode 1. Among the Q values ​​of state S1, which is the state corresponding to the current step (step 2), the maximum Q value is 7500. The transport operation corresponding to this Q value is a rearrangement operation to transport coil b1 from storage space 5 to storage space 11. Therefore, state S1 shown in FIG. 16(d) is updated to state S2 shown in FIG. 17(d). FIG. 17(d) is an example of information obtained by the processing of step S812 after the determination in step S810 is NO, for example.

[0133] Fig. 17(e) is a diagram showing an example of the transport operation at the time when step 2 of episode 1 is completed (the time when the state transitions from state S1 to state S2). Fig. 9C is a diagram showing an example of state S2 of the coil yard 210. The derivation unit 130 derives the work end time (7:51:56) of the transport work (receiving work to transport coil i1 to storage space 2) employed in the previous step, step 1 (state S0), as the work start time of the transport work (rearrangement work to transport coil b1 from storage space 5 to storage space 11) employed in the current step, step 2 (state S2). Furthermore, from FIG. 14B, the derivation unit 130 derives the work end time as the time (7:55:43) obtained by adding the work start time (7:51:56) to the travel time of the crane 240 from storage space 2 to storage space 5, which is 130 seconds, and the travel time of the crane 240 from storage space 5 to storage space 11, which is 97 seconds. In this specific example, a case is illustrated in which the start time of the reassignment work does not need to be taken into consideration, and therefore the work delay time is not derived. FIG. 17(e) shows an example of information obtained by the process of step S812 after the determination of NO in step S810.

[0134] 18 is a diagram illustrating an example of the processing of the third step (step 3) of the first episode (episode 1). When the processing of step 3 of episode 1 starts, the state S2 corresponding to step 3 and the action a for the state S2 are not registered in the Q table. FIG. 18(a) is a diagram showing an example of the coil C to be transported selected by the selection unit 120 in step 3 of episode 1. The work end time (7:55:43) in the previous step, step 2 (state S1), is 43 seconds after 7:55, which is the scheduled time of the dispensing work for the dispensing-scheduled coil e1 (the scheduled dispensing time in FIG. 12(c)). Therefore, the condition shown in (G) above is applied with priority over the other conditions shown in (A) to (F). Therefore, only the coil e1 to be transported is selected as the coil C to be transported. The destination of the coil C to be transported is the dispensing pit 230c. FIG. 18(a) is an example of information obtained by the processing of step S805 after the determinations of steps S813 and S804 are both NO.

[0135] FIG. 18(b) is a diagram showing an example of the immediate reward r (reward) derived by the derivation unit 130 in step 3 of episode 1. FIG. 18(b) is an example of information obtained by the processing of step S805, for example. Note that, as shown in FIG. 9C, the coil to be dispensed e1 is in the waiting area 210c. Therefore, the transport work of the coil to be dispensed e1 is a dispensing work from outside the dispensing area of ​​the coil to be dispensed C, and therefore is a transport work with a work category of 6 shown in FIG. 15(a). Therefore, the basic reward reward for the dispensing work of the coil to be dispensed e1 is base is 0.

[0136] In addition, the transportation cost -cost for the payout operation of the payout-scheduled coil e1 is the sum of the travel distance of the crane 240 from the storage space 11 to the storage space 7 and the travel distance from the storage space 7 to the payout pit 230c, and the cost weight w j The product of the delay time of the work completion time of the take-out work of the scheduled take-out coil e1 with respect to the scheduled time of the take-out work (scheduled take-out time), and the cost weight w for the receipt take-out delay time j The value is the sum of the product of and and multiplied by -1.

[0137] From Figure 15(b), the cost weight w j , cost weight for receipt / delivery delay time w j are 1 and 100, respectively. Also, from FIG. 14A, the travel distance of the crane 240 from the storage space 11 to the storage space 7 is 2100, and the travel distance from the storage space 7 to the dispensing pit 230c is 1800. Also, from FIG. 14B, the delay time of the end time of the dispensing operation of the coil to be dispensed e1 relative to the scheduled time of execution of the dispensing operation (scheduled dispensing time) is the sum of 100 seconds, which is the travel time from the storage space 11 to the storage space 7, 90 seconds, which is the travel time from the storage space 7 to the dispensing pit 230c, and the aforementioned delay time of 43 seconds, which is 233 seconds. Therefore, from FIG. 14A, the transportation cost -cost for the dispensing operation of the coil to be dispensed e1 is -27200 (= -1 × {1 × (2100 + 1800) + 100 × 233}). In addition, the additional reward for the dispensing work of the coil e1 to be dispensed is 0. From the above, the immediate reward for the action of transporting the dispensing coil e1 from the storage space 7 to the dispensing pit 230c is −27300 (=0−27300+0) according to equation (7).

[0138] Fig. 18(c) is a diagram showing an example of the Q table derived by the derivation unit 130 in step 3 of episode 1. Fig. 18(c) is an example of information obtained by the processing of step S806, for example.

[0139] FIG. 18(d) is a diagram showing an example of the result of updating the state in step 3 of episode 1. Among the Q values ​​of state S3, which is the state corresponding to the current step (step 3), the maximum Q value is -13600. The transport operation corresponding to this Q value is the dispensing operation of transporting the coil e1 to be dispensed from the storage space 7 to the dispensing pit 230c. Therefore, state S2 shown in FIG. 17(d) is updated to state S3 illustrated in FIG. 18(d). FIG. 18(d) is an example of information obtained by the processing of step S812 after a NO determination is made in step S810, for example.

[0140] Fig. 18(e) is a diagram showing an example of the transport operation at the time when step 3 of episode 1 is completed (the time when the state transitions from state S2 to state S3). Fig. 9D is a diagram showing an example of state S3 of the coil yard 210. As mentioned above, the operation delay time of the dispensing operation of the coil to be dispensed e1 is 233 seconds.

[0141] 19 is a diagram illustrating an example of the processing of the fourth step (step 4) of the first episode (episode 1). When the processing of step 4 of episode 1 starts, the state S3 corresponding to step 4 and the action a for that state S3 are not registered in the Q table. FIG. 19(a) is a diagram showing an example of coil C to be transported, selected by the selector 120 in step 4 of episode 1. Here, coil b2 is selected, which satisfies condition (A) among the conditions (A) to (G) described above. Furthermore, one of the storage space candidates is storage space 7 (see FIGS. 9D and 13). FIG. 19(a) is an example of information obtained by the processing of step S805 after the determinations of steps S813 and S804 are both NO.

[0142] 19(b) is a diagram showing an example of the immediate reward r (reward) derived by the derivation unit 130 in step 4 of episode 1. FIG. 19(b) is an example of information obtained by the process of step S805, for example. Fig. 19(c) is a diagram showing an example of the Q table derived by the derivation unit 130 in step 4 of episode 1. Fig. 19(c) is an example of information obtained by the processing of step S806, for example.

[0143] FIG. 19(d) is a diagram showing an example of the state update result in step 4 of episode 1. Among the Q values ​​of state S3, which is the state corresponding to the current step (step 4), the maximum Q value is 8000. The transport operation corresponding to this Q value is a rearrangement operation to transport coil b2 from storage space 9 to storage space 7. Therefore, state S3 shown in FIG. 18(d) is updated to state S4 shown in FIG. 19(d). FIG. 19(d) is an example of information obtained by the processing of step S812 after the determination in step S810 is NO, for example.

[0144] Fig. 19(e) is a diagram showing an example of the transport operation at the time when step 4 of episode 1 is completed (the time when the state transitions from state S3 to state S4). Fig. 9E is a diagram showing an example of state S4 of the coil yard 210.

[0145] 20 is a diagram illustrating an example of the processing of the fifth step (step 5) of the first episode (episode 1). When the processing of step 5 of episode 1 starts, state S4 corresponding to step 5 and action a for state S4 are not registered in the Q table. FIG. 20(a) is a diagram showing an example of the coil C to be transported selected by the selection unit 120 in step 5 of episode 1. The work end time (8:02:07) in the previous step, step 4 (state S3), is 127 seconds after 8:00, which is the scheduled time of the dispensing work for the coil to be dispensed e2 (the scheduled dispensing time in FIG. 12(c)). Therefore, the condition shown in (G) above is applied with priority over the other conditions shown in (A) to (F). Therefore, only the coil to be dispensed e2 is selected as the coil C to be transported. The destination of the coil to be dispensed C is the dispensing pit 230c. FIG. 20(a) is an example of information obtained by the processing of step S805 after the determinations of NO are made in steps S813 and S804.

[0146] Fig. 20(b) is a diagram showing an example of the immediate reward r (reward) derived by the derivation unit 130 in step 5 of episode 1. Fig. 20(b) is an example of information obtained by the processing of step S805, for example. Fig. 20(c) is a diagram showing an example of the Q table derived by the derivation unit 130 in step 5 of episode 1. Fig. 20(c) is an example of information obtained by the processing of step S806, for example.

[0147] FIG. 20(d) is a diagram showing an example of the result of updating the state in step 5 of episode 1. Among the Q values ​​of state S4, which is the state corresponding to the current step (step 5), the maximum Q value is -17550. The transport operation corresponding to this Q value is the dispensing operation of transporting the coil e2 to be dispensed from the storage space 10 to the dispensing pit 230c. Therefore, state S4 shown in FIG. 19(d) is updated to state S5 shown in FIG. 20(d). FIG. 20(d) is an example of information obtained by the processing of step S812 after step S810 is determined to be NO, for example.

[0148] FIG. 20(e) is a diagram showing an example of the transport operation at the end of step 5 of episode 1 (the point at which the state transitions from state S4 to state S5). FIG. 9F is a diagram showing an example of state S5 of the coil yard 210. The method for deriving the operation delay time for the delivery operation of coil e2 to be delivered is the same as the method for deriving the operation delay time for coil e1 to be delivered described in step 3 of episode 1, so a detailed description will be omitted. However, the operation delay time for the delivery operation of coil e2 to be delivered is 313 seconds. Therefore, the cumulative total of the receiving and delivery operation delay times in steps 1 to 5 of episode 1 is 546 seconds (= 233 + 313). The receiving and delivery operation delay time is the sum of the delay time of the receiving operation start time in the transport plan for coil C to be delivered relative to the scheduled execution time of the receiving operation and the delay time of the delivery operation end time in the transport plan for coil C to be delivered relative to the scheduled execution time of the delivery operation.

[0149] As shown in Figure 20(c), when step 5 of episode 1 is completed, the state in which all of the receiving and dispensing operations included in the transport plan have been completed is registered in the Q table. In this specific example, the state S0 indicating the state in which the receiving operation for the coil to be received i1 has been completed, the state S2 indicating the state in which the dispensing operation for the coil to be dispensed e1 has been completed, and the state S4 indicating the state in which the dispensing operation for the coil to be dispensed e2 has been completed are registered in the Q table. Therefore, the derivation unit 130 determines that the first episode (episode 1) has been completed (determines YES in step S813).

[0150] FIG. 21 is a diagram showing an overview of the transport work (coil C to be transported, transport source, and transport destination) performed in each state at the end of the first episode (episode 1) and an example of the immediate reward in each state. The total of the immediate rewards in each state at the end of the first episode (episode 1) is −13,000. At this point, there is no previous episode. Therefore, the derivation unit 130 determines not to end the learning in step S814 and updates the episode to the next episode (episode 2) in step S815. At this time, the derivation unit 130 returns the state of the coil yard 210 to the initial state S0 illustrated in FIG. 9A. Then, the process of step S803 is performed again. Note that, as described above, in this specific example, the random selection rate ε is 0, so the process of step S816 is not performed.

[0151] << <episode2>>> FIG. 22 is a diagram illustrating an example of the processing of the first step (step 1) of the second episode (episode 2). At the time when the processing of step 1 of episode 2 starts, the initial state S0 corresponding to step 1 and the action a for that state S0 are registered in the Q table, so a YES determination is made in step S804. Also, in this specific example, the random selection rate ε is 0, so the determination in step S807 is not made and the processing of step S808 is performed. In step S808, the selection unit 120 selects from the Q table the action (transportation work) corresponding to the largest Q value among the Q values ​​of the initial state S0, which is the state corresponding to the current step (step 1). Also, the next state S1 of the state S0 corresponding to the current step (step 1) and the action a for that state S1 are registered in the Q table. Therefore, a YES determination is made in step S810, and the processing of step S811 is performed.

[0152] Fig. 22(a) is a diagram showing an example of a Q table derived by the derivation unit 130 at the start of episode 2. Fig. 22(a) is the same as the Q table shown in Fig. 20(c) (the Q table derived in step 5, which is the final step of episode 1).

[0153] In FIG. 22(a), the maximum Q value among the Q values ​​of the initial state S0 is 9150. The transportation task corresponding to this Q value is the transportation task (receiving task) of transporting the to-be-received coil i1 from the receiving area 210a to the storage space 2. Therefore, the derivation unit 130 derives an immediate reward for this transportation task. As described with reference to FIG. 16(b), this immediate reward is 18300. Note that since this immediate reward (=18300) has already been derived in episode 1, the derivation unit 130 may store this immediate reward. In this case, the derivation unit 130 does not need to derive this immediate reward in episodes 2 and later. Furthermore, the maximum Q value among the Q values ​​of state S1, which is the state next to the initial state S0, is 7500. The derivation unit 130 uses these values ​​to update the Q value (Q[S0,(i1,2)]) for the transportation task. Specifically, from equation (11), the Q value (Q[S0,(i1,2)]) is updated to 17437.5 (=(1-0.5)×9150+0.5×(18300+0.99×7500)). Figure 22(b) is a diagram showing an example of a Q table after updating the Q table shown in Figure 22(a). Figure 22(b) is an example of information obtained by the processing of step S811, for example.

[0154] FIG. 22(c) is a diagram showing an example of the state update result in step 1 of episode 2. Among the Q values ​​of the initial state S0, which is the state corresponding to the current step (step 1), the maximum Q value is 17437.5. The transport operation corresponding to this Q value is the receiving operation of transporting the coil i1 to be received from the receiving pit 230a to the storage space 2. Therefore, the initial state S0 shown in FIG. 11 is updated to the state S1 illustrated in FIG. 22(c). FIG. 22(c) shows an example of information obtained by the process of step S812 after the determination of YES in step S810.

[0155] FIG. 22(d) is a diagram showing an example of the transport operation at the time when step 1 of episode 2 is completed (at the time when the state transitions from the initial state S0 to state S1). FIG. 22(d) shows an example of information obtained by the process of step S812 after the determination of YES in step S810. After this, the processing from step 2 onwards in episode 2 (repeated processing of S804 to S813) is carried out.

[0156] FIG. 23 is a diagram showing an example of information that can be obtained at the end of the second episode (episode 2). FIG. 23(a) is a diagram showing an example of a Q table derived by the derivation unit 130 in step 5, which is the final step of episode 2. Note that FIG. 23(a) is a diagram corresponding to FIG. 20(b). FIG. 23(b) is a diagram showing an outline of the transport work (coil C to be transported, transport source, transport destination) performed in each state at the end of episode 2, and an example of the immediate reward in each state. Note that FIG. 23(b) is a diagram corresponding to FIG. 21.

[0157] FIG. 24 is a diagram showing an example of information that can be obtained when the ninth episode (episode 9) ends. FIG. 24(a) is a diagram illustrating an example of a Q table derived by the derivation unit 130 in step 7, which is the final step of the ninth episode (episode 9). Note that FIG. 24(a) corresponds to FIG. 20(b) and FIG. 23(a). FIG. 24(b) is a diagram illustrating an example of the transport operation at the end of step 7 of episode 9 (at the time of the transition from state S6 to state S7). Note that FIG. 24(b) corresponds to FIG. 20(e). FIG. 24(c) is a diagram illustrating an overview of the transport operation (coil C to be transported, transport source, and transport destination) performed in each state at the end of episode 9, and an example of the immediate reward in each state. Note that FIG. 24(c) corresponds to FIG. 21 and FIG. 23(b). In this specific example, the solution was not updated in episode 10, the episode following episode 9 (the solution at the end of episode 10 was the same as the solution at the end of episode 9). Therefore, in step S817, the output unit 140 outputs, for example, the information shown in FIG. 24(b) as an example of the transport information.

[0158] <Results of operation plan creation> 25 is a diagram showing an example of the conditions for creating an operation plan and the results of the creation. Here, an operation plan created by an operator based on their experience in actual operation is taken as a comparative example. An operation plan created by the method of this embodiment using data obtained during the actual operation is taken as an example of the invention. 25(a) is a diagram showing the calculation conditions (problem scale) for the operation plan. The total number of storage spaces is the total number of storage spaces in the coil yard 210. The initial inventory quantity is the total number of coils C placed in the coil yard 210 when the transportation plan was created.

[0159] FIG. 25(b) is a diagram showing the results of the invention example and the comparative example. The total travel distance is the total travel distance of the transport means (crane). The total number of transfers is the total number of transfer operations. The total receiving time is the total time from the start to the end of the receiving operation. The total pick-up time is the total time from the start to the end of the pick-up operation. The total number of constraint violations is the total number of transport operations that do not satisfy constraints, including the destination constraints described with reference to Figure 6. As shown in FIG. 25(a), it can be seen that by creating a transportation plan using the method of this embodiment, it is possible to reduce the workload, work time, and constraint violations compared to a transportation plan created based on the experience of workers.

[0160] <Summary> As described above, in this embodiment, the plan creation device 100 selects transport objects to be transported from among transport objects that can be transported by the transport means, and derives transport information, which is information that can at least identify the transport order and destination of the selected transport objects, by performing reinforcement learning. The selection of transport objects to be transported includes selection based on predetermined rules as conditions for the transport objects to be transported. Therefore, it is possible to prevent an increase in the time required to create a transport plan. Therefore, for example, even when there is a large-scale transport request involving a large number of transport objects, an efficient transport plan can be created.

[0161] Furthermore, in this embodiment, the plan creation device 100 selects candidate destinations for the transported object to be transported in the transport operation, and derives the destination of the transported object from the candidates by performing reinforcement learning. The selection of candidate destinations for the transported object to be transported in the transport operation includes selection based on predetermined rules as conditions for destinations to which the transported object can be transported. Therefore, an increase in the time required to create a transport plan can be further suppressed.

[0162] Furthermore, in this embodiment, the plan creation device 100 derives an immediate reward for the action of transporting the transport object using at least one of a positive reward and a negative reward, the value of which is determined for each of a plurality of predetermined categories according to the content of the transport operation, and derives transport information based on the derived immediate reward. Therefore, an immediate reward according to the content of the transport operation can be used. Therefore, it is possible to create a transport plan that can transport the transport object more efficiently while suppressing an increase in the time required to create the transport plan.

[0163] Here, the negative reward may be determined based on at least one of the following: the degree of workload of the transportation work; the degree of delay from the scheduled time of at least one of the receiving work and the unloading work; and whether the transportation work is feasible. For example, when the negative reward is determined based on the degree of workload of the transportation work, a transportation plan can be created that can prevent an increase in workload. Furthermore, when the negative reward is determined based on the degree of delay from the scheduled time of at least one of the receiving work and the unloading work, a transportation plan can be created that can prevent delays of the receiving work and the transportation work from the scheduled time of the receiving work and the unloading work, which dynamically change depending on the transported object. Furthermore, when the negative reward is determined based on whether the transportation work is feasible, the creation of an unfeasible transportation plan can be prevented.

[0164] The value of the positive reward may also be determined depending on whether the transportation task is a specific transportation task. The specific transportation task includes at least one of the above-mentioned unloading task, transportation task required to perform the unloading task, receiving task, and transportation task required to perform the receiving task. In this way, a transportation plan can be created in which receiving task and unloading task are performed with priority.

[0165] Furthermore, in this embodiment, the plan creation device 100 derives an immediate reward for the action of transporting the selected transport object based on the result of simulating the action of transporting the transport object. Therefore, it is possible to derive an immediate reward according to how the transport object is transported. For example, by deriving an immediate reward for the action of transporting the transport object based on at least one of the start time and end time of the transport of the transport object, it is possible to prevent the creation of a transportation plan in which the transport starts or ends at a time later than a predetermined time.

[0166] In this embodiment, the predetermined rules as conditions for the transported object to be transported include selecting the transported object or an object that needs to be transported in order to perform the receiving operation for the transported object when the time when the receiving operation for the transported object to be performed is later than the scheduled time of the receiving operation. Therefore, when the scheduled time of the receiving operation is later, a transport plan can be created in which the receiving operation is given priority.

[0167] In this embodiment, the predetermined rules as conditions for the transported object to be transported include selecting the transported object or the transported object that needs to be transported in order to perform the transported object's removal operation when the time when the removal operation of the transported object to be removed is to be performed later than the scheduled time of the removal operation. Therefore, a transport plan can be created in which the removal operation is given priority when the scheduled time of the removal operation is to be delayed.

[0168] Furthermore, in this embodiment, when an action a for a transport object in a certain state S has not been derived in reinforcement learning, the plan creation device 100 selects a transport object to be a target of a transport operation based on predetermined rules as conditions for the transport object to be transported. Therefore, reinforcement learning can be performed after reducing the problem scale using a rule-based method. Therefore, it is possible to create a transportation plan that can transport the transport object more efficiently while suppressing an increase in the time required to create the transportation plan.

[0169] In the present embodiment, the plan creation device 100 derives, as the transportation information, information that can identify at least the time when the transportation work of the transported object is to be performed and the destination of the transported object by performing reinforcement learning. Therefore, it is possible to create a transportation plan in which the performance time of each transportation work is determined.

[0170] (Other embodiments) The above-described embodiments of the present disclosure can be realized by a computer executing a program. A computer-readable recording medium on which the program is recorded and a computer program product such as the program can also be applied as embodiments of the present disclosure. Examples of recording media that can be used include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, magnetic tapes, non-volatile memory cards, and ROMs. Furthermore, the above-described embodiments of the present disclosure are merely examples of specific embodiments for carrying out the present disclosure, and the technical scope of the present disclosure should not be interpreted as being limited by these. In other words, the present disclosure can be embodied in various forms without departing from its technical concept or main features.

[0171] The disclosure of the above embodiment can be implemented as follows, for example. [Disclosure 1] A planning device that performs processing to create a plan for a transport operation that includes at least one of a receiving operation that is a transport operation of transported objects to a storage yard and a removal operation that is a transport operation of transported objects from the storage yard, a selection unit that selects an object to be transported from among objects that can be transported by the transport means; a derivation unit that derives transport information, which is information that can identify at least the transport order and destination of the transported objects selected by the selection unit, by performing reinforcement learning in which a space including the storage area is an environment, the arrangement of the transported objects in the space including the storage area is a state, and transporting the transported objects is an action; Equipped with The selection of the transported object to be the target of the transport operation by the selection unit includes selection based on a rule that is predetermined as a condition for the transported object to be transported. [Disclosure 2] the selection unit further selects a candidate destination for the transported object that is the target of the transport operation; The selection of the candidate destinations of the object to be transported includes selection based on predetermined rules as conditions for the destinations to which the object can be transported, The plan creation device according to Disclosure 1, wherein the derivation unit derives a destination of the object selected by the selection unit from the candidates by performing the reinforcement learning. [Disclosure 3] The plan creation device according to Disclosure 1 or 2, wherein the derivation unit derives an immediate reward for the action of transporting the transported object selected by the selection unit using at least one of a positive reward and a negative reward, the values ​​of which are determined for each of a plurality of predetermined categories according to the content of the transport work, and derives the transport information based on the derived immediate reward. [Disclosure 4] the derivation unit derives an immediate reward for the action of transporting the object selected by the selection unit, using the negative reward; The plan creation device described in Disclosure 3, wherein the negative reward is determined according to at least one of the degree of workload of the transportation work, the degree of delay from the scheduled time of execution of at least one of the receiving work and the unloading work, and whether or not the transportation work is feasible. [Disclosure 5] the derivation unit derives an immediate reward for the action of transporting the object selected by the selection unit, using the positive reward; the value of the positive reward is determined depending on whether the transportation task is a specific transportation task; The planning device described in Disclosure 3 or 4, wherein the specific transportation work includes at least one of the withdrawal work, the transportation work required to perform the withdrawal work, the receiving work, and the transportation work required to perform the receiving work. [Disclosure 6] The plan creation device described in any one of Disclosures 3 to 5, wherein the derivation unit derives an immediate reward for the action of transporting the transport object based on the results of simulating the action of transporting the transport object selected by the selection unit. [Disclosure 7] The plan creation device described in Disclosure 6, wherein the derivation unit derives at least one of the start time and end time of transporting the transported object by simulating the action of transporting the transported object selected by the selection unit, and derives an immediate reward for the action of transporting the transported object based on the derived time. [Disclosure 8] the plan includes a plan for the receiving operation; The planning device described in any one of Disclosures 1 to 7, wherein the predetermined rules as conditions for the transported object to be transported include selecting the transported object or the transported object that needs to be transported in order to perform the reception work for the transported object when the time at which the reception work for the transported object to be subjected to the reception work is performed is later than the scheduled time at which the reception work is performed. [Disclosure 9] The plan includes a plan for the removal operation, A planning device described in any one of Disclosures 1 to 8, wherein the rules predetermined as conditions for the transported object to be transported include selecting the transported object or the transported object that needs to be transported in order to perform the removal work on the transported object when the time at which the removal work on the transported object to be removed is performed is later than the scheduled time at which the removal work is performed. [Disclosure 10] The planning device described in any one of Disclosures 1 to 9, wherein when the action for the transported object in the state has not been derived in reinforcement learning, the selection unit selects a transported object to be the target of the transport operation from among the transported objects in the state that can be transported by the transport means based on rules that are predetermined as conditions for the transported object to be transported. [Disclosure 11] The planning device according to Disclosure 10, wherein the reinforcement learning is Q-learning. [Disclosure 12] 12. The plan creation device according to any one of Disclosures 1 to 11, wherein the plan is a plan for a transport operation including the receiving operation and the unloading operation. [Disclosure 13] an acquisition unit that acquires information including attribute information, inventory information, and at least one of receipt timing information and delivery timing information; the attribute information is information that can identify at least the attributes of the transportable object, The inventory information is information that can at least identify the location of the transported item in the storage area, the receiving timing information is information that can at least identify the order of the receiving operation of the transported objects that are the targets of the receiving operation, the delivery timing information is information that can at least identify the order of the delivery operation of the transported objects that are the targets of the delivery operation, The planning device described in any one of Disclosures 1 to 12, wherein the selection unit selects an object to be transported as a target for the transport operation from among objects that can be transported by the transport means based on the information acquired by the acquisition unit. [Disclosure 14] the reception timing information is information that can at least identify a scheduled time when the reception work of the transported object to be performed is to be performed, the delivery timing information is information that can at least identify a scheduled time when the delivery operation of the transported object that is the target of the delivery operation is to be performed, The plan creation device described in Disclosure 13, wherein the derivation unit derives transportation information, which is information that can at least identify the time when the transportation work of the transported object selected by the selection unit will be performed and the destination of the transported object, by performing the reinforcement learning. [Disclosure 15] A planning method for performing processing to create a plan for transport operations including at least one of receiving operations, which are operations for transporting transported objects to a storage yard, and unloading operations, which are operations for transporting transported objects from a storage yard, comprising: a selection step of selecting an object to be transported from among objects that can be transported by the transport means; a derivation procedure for deriving transport information, which is information capable of identifying at least the transport order and destination of the transported objects selected by the selection procedure, by performing reinforcement learning in which a space including the storage area is an environment, the arrangement of the transported objects in the space including the storage area is a state, and transporting the transported objects is an action; Equipped with The planning method, wherein the selection of the transported object to be the target of the transport operation by the selection procedure includes selection based on a rule that is predetermined as a condition for the transported object to be transported. [Disclosure 16] A program for causing a computer to function as each unit of the plan creation device according to any one of Disclosures 1 to 14. [Explanation of symbols]

[0172] 100 Planning device 110 Acquisition Department 120 Selection Section 130 Derivation part 140 Output section 210 coil yards 210a Reception Area 210b Payout Area 210c Waiting Area 220 Processing Line 230a Receiving pit 230b Receiving and discharging pit 230c Discharge Pit C11~C25, C31~C39 coils P1~P4 Positions through which the crane passes when moving SP1~SP2 Coil storage space b1~b2 coil e1~e2 Coils to be paid out i1 Coils to be received

Claims

1. A planning device that performs processing to create a plan for a transport operation that includes at least one of a receiving operation that is a transport operation of transported objects to a storage yard and a removal operation that is a transport operation of transported objects from the storage yard, a selection unit that selects an object to be transported from among objects that can be transported by the transport means; a derivation unit that derives transport information, which is information that can identify at least the transport order and destination of the transported objects selected by the selection unit, by performing reinforcement learning in which a space including the storage area is an environment, the arrangement of the transported objects in the space including the storage area is a state, and transporting the transported objects is an action; Equipped with The selection of the transported object to be the target of the transport operation by the selection unit includes selection based on a rule that is predetermined as a condition for the transported object to be transported.

2. the selection unit further selects a candidate destination for the transported object that is the target of the transport operation; The selection of the candidate destinations of the object to be transported includes selection based on predetermined rules as conditions for the destinations to which the object can be transported, The plan creation device according to claim 1 , wherein the derivation unit derives the destination of the transport object selected by the selection unit from the candidates by performing the reinforcement learning.

3. 3. The plan creation device according to claim 1, wherein the derivation unit derives an immediate reward for the action of transporting the transported object selected by the selection unit using at least one of a positive reward and a negative reward, the values ​​of which are determined for each of a plurality of predetermined categories according to the content of the transport work, and derives the transport information based on the derived immediate reward.

4. the derivation unit derives an immediate reward for the action of transporting the object selected by the selection unit, using the negative reward; 4. The plan creation device according to claim 3, wherein the negative reward is determined according to at least one of the degree of workload of the transportation work, the degree of delay from the scheduled time of execution of at least one of the receiving work and the unloading work, and whether or not the transportation work is feasible.

5. the derivation unit derives an immediate reward for the action of transporting the object selected by the selection unit, using the positive reward; the value of the positive reward is determined depending on whether the transportation task is a specific transportation task; 4. The plan creation device according to claim 3, wherein the specific transport work includes at least one of the removal work, a transport work required to perform the removal work, the receiving work, and a transport work required to perform the receiving work.

6. The plan creation device according to claim 3 , wherein the deriving unit derives an immediate reward for the action of transporting the object based on a result of simulating the action of transporting the object selected by the selecting unit.

7. 7. The plan creation device according to claim 6, wherein the derivation unit derives at least one of a start time and an end time of transporting the object by simulating the action of transporting the object selected by the selection unit, and derives an immediate reward for the action of transporting the object based on the derived time.

8. the plan includes a plan for the receiving operation; 3. The planning device of claim 1, wherein the predetermined rules as conditions for the transported object to be transported include selecting the transported object or the transported object that needs to be transported in order to perform the reception work for the transported object when the time at which the reception work for the transported object to be performed is later than the scheduled time at which the reception work is performed.

9. The plan includes a plan for the removal operation, The planning device of claim 1 or 2, wherein the predetermined rules as conditions for the transported object to be transported include selecting the transported object or the transported object that needs to be transported in order to perform the removal work on the transported object when the time at which the removal work on the transported object to be removed is performed is later than the scheduled time at which the removal work is performed.

10. 3. The planning device according to claim 1, wherein when the behavior for the transported object in the state has not been derived in reinforcement learning, the selection unit selects the transported object to be the target of the transport operation from among the transported objects in the state that can be transported by the transport means based on rules that are predetermined as conditions for the transported object to be transported.

11. The plan creation device according to claim 10 , wherein the reinforcement learning is Q-learning.

12. The plan creation device according to claim 1 or 2, wherein the plan is a plan for a transport operation including the receiving operation and the unloading operation.

13. an acquisition unit that acquires information including attribute information, inventory information, and at least one of receipt timing information and delivery timing information; the attribute information is information that can identify at least the attributes of the transportable object, The inventory information is information that can at least identify the location of the transported item in the storage area, the receiving timing information is information that can at least identify the order of the receiving operation of the transported objects that are the targets of the receiving operation, the delivery timing information is information that can at least identify the order of the delivery operation of the transported objects that are the targets of the delivery operation, 3. The plan creation device according to claim 1, wherein the selection unit selects an object to be transported as a target for the transport operation from among objects that can be transported by the transport means based on the information acquired by the acquisition unit.

14. the reception timing information is information that can at least identify a scheduled time when the reception work of the transported object to be performed is to be performed, the delivery timing information is information that can at least identify a scheduled time when the delivery operation of the transported object that is the target of the delivery operation is to be performed, 14. The plan creation device according to claim 13, wherein the derivation unit derives transportation information, which is information capable of identifying at least the time when transportation work for the transported object selected by the selection unit will be performed and the destination of the transported object, by performing the reinforcement learning.

15. A planning method for performing processing to create a plan for transport operations including at least one of receiving operations, which are operations for transporting transported objects to a storage yard, and unloading operations, which are operations for transporting transported objects from a storage yard, comprising: a selection step of selecting an object to be transported from among objects that can be transported by the transport means; a derivation procedure for deriving transport information, which is information capable of identifying at least the transport order and destination of the transported objects selected by the selection procedure, by performing reinforcement learning in which a space including the storage area is an environment, the arrangement of the transported objects in the space including the storage area is a state, and transporting the transported objects is an action; Equipped with The planning method, wherein the selection of the transported object to be the target of the transport operation by the selection procedure includes selection based on a rule that is predetermined as a condition for the transported object to be transported.

16. A program for causing a computer to function as each unit of the plan creation device according to claim 1 or 2.

Images