Control device, control method, and program

Abstract

A control device 1X primarily comprises: an operation planning means 15X; and a determination means 16X. The operation planning means 15X determines a robot operation plan related to the work of sorting a plurality of objects into a plurality of sorting receptacles. The determination means 16X determines whether the sorting receptacles have been refilled. If the sorting receptacles have been refilled, the operation planning means 15X updates the operation plan based on the arrangement of the refilled sorting receptacles.

Description

[Technical field]

[0001] The present invention relates to the technical field of a control device, a control method, and a storage medium that perform processing related to a task to be performed by a robot. [Background technology]

[0002] A control method has been proposed for controlling a robot required to execute a task given to the robot. For example, Patent Document 1 discloses a robot control device that, when a robot with hands grasps multiple objects and stores them in a container, determines a combination of the order in which the hands grasp the objects, and determines the order of the objects to be stored based on an index calculated for each combination. Patent Document 2 discloses a system that causes a robot to pack loose items into a sorting pallet according to a packing plan for the loose items. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-051684 A [Patent Document 2] JP 2010-202291 A Summary of the Invention [Problem to be solved by the invention]

[0004] The task of sorting multiple types of items contained in boxes such as cardboard by type or lot involves multiple tasks such as removing the items from the boxes, moving them to sorting containers (trays) according to the type of item, etc., and transporting the trays, making it difficult to formulate an operating plan for automating the task using robots.

[0005] In view of the above-mentioned problems, one object of the present invention is to provide a control device, a control method, and a storage medium that are capable of causing a robot to suitably execute a task relating to sorting items. [Means for solving the problem]

[0006] One aspect of the control device is Multiple items are classified into multiple categories based on the classification of the items. No. 1 A motion planning means for determining a motion plan for the robot with respect to a pick-and-place operation for moving the robot to a sorting container; The above No. 1 Sorting Container Second sorting container with misalignment and a determination means for determining whether or not the ink has been replenished; The motion planning means Second If the sorting container is refilled, the above after refilling No. 1 Sorting Container and the second sorting container and updating the motion plan based on the arrangement of the target object.

[0007] One aspect of the control method includes: The computer Multiple items are classified into multiple categories based on the classification of the items. No. 1 determining a motion plan for the robot with respect to a pick-and-place operation for moving the items to the sorting container; The above No. 1 Sorting Container Second sorting container with misalignment Determine whether or not the item has been replenished; The above Second If the sorting container is refilled, the above after refilling No. 1 Sorting Container and the second sorting container and updating the motion plan based on the arrangement of the motion vectors.

[0008] One aspect of the program is Multiple items are classified into multiple categories based on the classification of the items. No. 1 determining a motion plan for the robot with respect to a pick-and-place operation for moving the items to the sorting container; The above No. 1 Sorting Container Second sorting container with misalignment Determine whether or not the item has been replenished; The above Second If the sorting container is refilled, the above after refilling No. 1 Sorting Container and the second sorting containerThe program causes a computer to execute a process of updating the operation plan based on the arrangement of the Effect of the Invention

[0009] The task of sorting items can be suitably performed by a robot. [Brief description of the drawings]

[0010] [Figure 1] 1 shows an example of the configuration of a sorting control system in a first embodiment. [Diagram 2] 2 illustrates an example of a hardware configuration of a control device. [Diagram 3] This is an example of an overhead view of a work space for loose sorting. [Figure 4] 3 is an example of a functional block of a control device. [Diagram 5] 13 shows an example of a data structure of application information. [Figure 6] 3 is an example of a functional block of a motion planning unit. [Figure 7] An example of an overhead view of a pick-and-place workspace that simplifies the bulk sorting work shown in Figure 3 is shown. [Figure 8] 4 is an example of a flowchart illustrating an overview of a robot control process executed by the control device in the first embodiment. [Figure 9] 13 shows an example of the configuration of a sorting control system in a modified example. [Figure 10] 13 is an example of a block configuration of a control device in the second embodiment. [Figure 11] 13 is an example of a flowchart executed by the control device in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of a control device, a control method, and a storage medium will be described with reference to the drawings.

[0012] First Embodiment (1) System Configuration 1 shows the configuration of a sorting control system 100 according to the first embodiment. The sorting control system 100 is a system that automatically or semi-automatically executes the task of sorting items (products) such as medicines delivered to a warehouse by type or by lot (also called "bulk sorting"), and mainly comprises a control device 1, a measuring device 4, and a controlled device 5. The control device 1, the measuring device 4, and the controlled device 5 communicate data via a communication network 3.

[0013] The control device 1 supplies a control signal "S2" required to execute the bulk sorting to the controlled device 5 such as the robot 51 based on the measurement signal "S1" supplied from the measuring device 4. In this case, the control device 1 sets a task to be executed by the robot 51 (also called a "target task"), and formulates an operation plan for the robot 51 to execute the set target task based on temporal logic. In this case, the control device 1 converts the target task into a sequence for each time step (time interval) of the task that the robot 51 can accept. Then, the control device 1 controls the robot 51 by supplying a control signal S2 representing the generated operation sequence to the robot 51. Hereinafter, a task (command) obtained by breaking down the target task into units that the robot 51 can accept is also called a "subtask", and a sequence of subtasks that the robot 51 should execute to achieve the target task is called an "operation sequence". Note that in this embodiment, a process of formulating an operation plan based on temporal logic will be described as an example, but the method is not limited to formulating an operation plan based on temporal logic, and the control device 1 may formulate an operation plan by any other method.

[0014] The measuring device 4 is a sensor that detects the state in the work space for bulk sorting. The measuring device 4 includes, for example, a camera 41 and a BCR (Bar-Code Reader) 42. The measuring device 4 supplies the generated measurement signal S1 to the control device 1 via the communication network 3. The measuring device 4 may include a self-propelled or flying sensor (including a drone) that moves in the work space. The measuring device 4 may also include a sensor provided in a controlled device 5 such as a robot 51, and a sensor provided in another object in the work space. The measuring device 4 may also include a sensor that detects sound in the work space. In this way, the measuring device 4 may include various sensors that detect the state in the work space and are provided at any location.

[0015] The controlled device 5 is provided in the work space for bulk sorting, and is a device controlled based on a control signal S2 and the like supplied from the control device 1. In the present embodiment, as an example, the controlled device 5 has a robot 51, a conveying machine (conveyor) 52 for conveying items in a warehouse, and a sorting container refilling mechanism 53 for refilling sorting containers such as trays for containing items sorted by the robot 51. A plurality of robots 51 are present in the work space, and they cooperate with each other to execute a task related to bulk sorting as necessary. The robot 51 may be a vertical articulated robot, a horizontal articulated robot, or any other type of robot. The robot 51 may supply a status signal indicating the status of the robot 51 to the control device 1. This status signal may be an output signal of a sensor that detects the status (position, angle, etc.) of the entire robot 51 or a specific part such as a joint, or may be a signal generated by a control unit of the robot 51 indicating the progress status of the subtask of the robot 51.

[0016] The configuration of the sorting control system 100 shown in FIG. 1 is an example, and various modifications may be made to the configuration. For example, the control device 1 may perform data communication with the measuring device 4 and the controlled device 5 by direct wireless or wired communication without going through the communication network 3. The control device 1 and the robot 51 may be configured as one unit. The control device 1 may also be configured from a plurality of devices. In this case, the plurality of devices that configure the control device 1 transmit and receive information required to execute pre-assigned processing between these plurality of devices. The robot 51 may also incorporate at least some or all of the functions of the control device 1.

[0017] (2) Hardware Configuration 2 shows a hardware configuration of the control device 1. The control device 1 includes, as hardware, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 10.

[0018] The processor 11 functions as a controller (arithmetic unit) that performs overall control of the control device 1 by executing a program stored in the memory 12. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0019] The memory 12 is composed of various types of volatile and non-volatile memories such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, etc. The memory 12 also stores a program for executing the processes executed by the control device 1. The memory 12 also has application information IA and sorting information IB.

[0020] The application information IA is various information necessary to generate an operation sequence, which is a sequence to be executed by the robot 51, from a target task. The sorting information IB is information related to sorting of items, for example, information linking the item's identification information (ID) read from the barcode attached to each item with the sorting classification of the product. The above-mentioned classification is based on whether the items can be sorted into the same sorting container, for example, classification based on the type or lot of the item. Note that, if the sorting container has an ID (identification information), the sorting information IB may be information linking the item's ID read from the barcode with the ID of the sorting container into which the item is to be sorted.

[0021] Some of the information stored in the memory 12 may be stored in one or more external storage devices capable of communicating with the control device 1, or in a storage medium such as a flash memory that is removable from the control device 1.

[0022] The interface 13 is an interface for electrically connecting the control device 1 to other devices. These interfaces may be wireless interfaces such as a network adapter for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0023] The hardware configuration of the control device 1 is not limited to the configuration shown in Fig. 2. For example, the control device 1 may be connected to or built-in at least one of a display device, an input device, and a sound output device.

[0024] (3) Overview of bulk sorting Fig. 3 is an overhead view of a work space, which is a schematic representation of an example of bulk sorting by the sorting control system 100. Fig. 3 shows an operation of sorting items 7 (7A-7G, ...) taken out of a packing box 6, in which multiple types of items to be sorted are packed, into multiple sorting containers 8 (8A-8D, ...) in a warehouse where the packing box 6 is delivered from a manufacturer or the like. The camera 41 is installed so that the work space is included in its shooting range. The camera 41 may be a plurality of cameras with different shooting ranges, each of which may take a share of the location required for recognizing the state of the work space and shoot the location.

[0025] In the work space shown in Fig. 3, there are a first conveying machine 52A, a second conveying machine 52B, and a third conveying machine 52C, which are one embodiment of the conveying machine 52. In Fig. 3, arrows are added to the first conveying machine 52A, the second conveying machine 52B, and the third conveying machine 52C to additionally indicate the conveying direction.

[0026] The first conveying machine 52A is, for example, a belt conveyor, and conveys the article 7 (7A to 7G, ...) taken out of the packing box 6 by the worker or the robot in the direction of the movable range "RA" of the robot arm 51A or the movable range "RB" of the robot arm 51B. The first conveying machine 52A is provided with a BCR 42 that reads the barcode of each article before it reaches the movable range of the robot arms 51A and 51B, and each article 7 taken out of the packing box 6 is placed on the first conveying machine 52A so that the side with the barcode is facing upward. The conveying speed of the first conveying machine 52A may be a constant speed, or may be adjusted based on the control of the control device 1 or the like. For example, when the robot arm 51A and the robot arm 51B transfer the article 7, the first conveying machine 52A may be adjusted to be temporarily stopped or decelerated during the transfer operation based on the control of the control device 1. The first conveyance machine 52A may be adjusted to temporarily stop or decelerate not only during the above-mentioned transfer period but also during the entire operation period of the robot arm.

[0027] The second conveying machine 52B is a material handling machine that conveys the sorting containers 8 containing the sorted articles 7 to an area where shelves containing the sorting containers 8 are present.

[0028] The third conveying machine 52C conveys the item 7 that requires exceptional processing in the direction in which the movable range of the robot arm 51C exists. For example, the item 7 without a barcode (e.g., the item 7D) or the item 7 that does not have registered attribute information such as size and requires inspection processing is conveyed from the first conveying machine 52A to the third conveying machine 52C. Note that the work of conveying such an item 7 to the third conveying machine 52C may be performed by the robot arm 51B, may be performed by a machine other than the robot arm 51B, or may be performed manually. Similarly, the process of attaching a barcode or inspecting the item 7 conveyed to the third conveying machine 52C may be performed by the robot arm 51C, may be performed by a machine other than the robot arm 51C, or may be performed manually.

[0029] In addition, a sorting container table 54 on which the sorting containers 8 are placed is provided between the first conveying machine 52A and the second conveying machine 52B. The sorting container table 54 can hold up to four sorting containers 8, and the sorting containers 8 (8A-8D) used for sorting by the robot arms 51A and 51B are placed on the sorting container table 54. Similarly, a sorting container table 55 is provided between the third conveying machine 52C and the second conveying machine 52B.

[0030] The sorting containers 8 placed on the sorting container table 54 contain items 7 of the same type or the same lot for the same sorting container 8. The sorting containers 8 are transported to the second conveying machine 52B when a predetermined condition is satisfied. Specifically, when there is a sorting container 8 that has reached a predetermined reference state (for example, a state in which further items 7 cannot be accommodated) or when it becomes necessary to place an additional sorting container 8 on the sorting container table 54, the sorting container 8 is transported from the sorting container table 54 to the second conveying machine 52B. This transport operation may be performed by the sorting container table 54, or by the robot arm 51A or the robot arm 51B. In the former case, the sorting container table 54 has, for example, a mechanism that is individually slidable for each sorting container 8, and moves the specified sorting container 8 to the second conveying machine 52B by sliding the specified sorting container 8 in the direction of the second conveying machine 52B based on a control signal S2 supplied from the control device 1. In the latter case, the robot arm 51A or the robot arm 51B moves the target sorting container 8 to the second conveying machine 52B by a push-out operation or a pick-up place, based on a control signal S2 supplied from the control device 1.

[0031] The sorting containers 8 used by the robot arm 51C for sorting are also placed on the sorting container table 55. Then, the robot arm 51C places items 7 that have been subjected to exceptional processing, such as attaching a barcode and / or inspection processing, on the sorting containers 8 on the sorting container table 55. Note that since there are only a small number of items 7 that are subjected to exceptional processing, for example, one item 7 is used per sorting container 8. Therefore, the sorting containers 8 containing the items 7 are transported to the second conveying machine 52B by the robot arm 51C or the sorting container table 55.

[0032] The sorting container replenishing mechanism 53 is a mechanism that replenishing sorting containers 8 placed on the sorting container tables 54 and 55. For example, when a sorting container 8 on either the sorting container table 54 or the sorting container table 55 is transported to the second conveying machine 52B, the sorting container replenishing mechanism 53 supplies replenishing sorting containers 8 to the sorting container table 54 or the sorting container table 55 from which the sorting container 8 was transported. In this case, the sorting container replenishing mechanism 53 has, for example, a mechanism that holds the replenishing sorting containers 8, and a mechanism that supplies the sorting containers 8 by pushing (sliding) the replenishing sorting containers 8 to the destination sorting container table 54 or sorting container table 55. In this case, for example, when any of the sorting containers 8B to 8D in Fig. 3 is carried out, the sorting container refill mechanism 53 supplies a refill sorting container 8 to the sorting container table 54, thereby pushing the sorting container 8A to the position of the sorting container 8B in Fig. 3, and disposing the refill sorting container 8 in the position where the sorting container 8A was located. The sorting container refill mechanism 53 may be configured to receive the refill sorting containers 8 from a conveyor that transports the refill sorting containers 8.

[0033] The robot arm 51A and the robot arm 51B are provided at positions facing each other with the first conveyance machine 52A in between. In Fig. 3, the movable range RA of the robot arm 51A and the movable range RB of the robot arm 51B are indicated by dashed circles. The robot arm 51A and the robot arm 51B are examples of the "first robot" and the "second robot", respectively.

[0034] As shown in FIG. 3, the movable range RA of the robot arm 51A and the movable range RB of the robot arm 51B overlap partially on the first conveying machine 52A, and are provided at positions that cover the ranges on the first conveying machine 52A that are outside the movable ranges of each other. This allows at least one of the robot arm 51A and the robot arm 51B to grasp the article 7 on the first conveying machine 52A. Specifically, the articles 7A, 7C, 7F, etc. conveyed by the first conveying machine 52A are outside the movable range RA even when they are conveyed to the vicinity of the front of the robot arm 51A (here, the direction of the imaginary perpendicular line drawn from the position of the robot arm 51A to the first conveying machine 52A). On the other hand, these articles are within the movable range RB when they are conveyed to the vicinity of the front of the robot arm 51B. Furthermore, even if the items 7B and 7G transported by the first conveyance machine 52A are transported near the front of the robot arm 51B (here, in the direction of an imaginary perpendicular line drawn from the position of the robot arm 51B to the first conveyance machine 52A), they are outside the movable range RB. On the other hand, when these items are transported near the front of the robot arm 51A, they are within the movable range RA.

[0035] In addition, at least one of the robot arm 51A and the robot arm 51B is capable of placing articles 7 on the sorting containers 8A-8D placed on the sorting container stage 54. In the example of Fig. 3, the sorting containers 8C and 8D are within the movable range RA of the robot arm 51A but outside the movable range RB of the robot arm 51B, and the sorting containers 8A and 8B are outside the movable range RA of the robot arm 51A but within the movable range RB of the robot arm 51B.

[0036] In this way, the movable range RA and the movable range RB overlap, so that the robot arms 51A and 51B can transfer the article 7 to each other. In detail, when one robot arm has an article 7 to be stored in a sorting container 8 that the one robot arm cannot reach and the article 7 is outside the movable range of the other robot arm, the one robot arm moves the article 7 into the movable range of the other robot arm. In this case, the other robot arm moves the article 7 moved by the one robot arm to an appropriate sorting container 8 by pick-and-place. The control device 1 incorporates such an operation into the operation plan of the robot arms 51A and 51B, so that the robot arms 51A and 51B can perform a sorting operation so that the article 7 placed at any position on the first conveying machine 52A is placed in an appropriate sorting container 8.

[0037] A specific example of the above-mentioned transfer will be described with reference to FIG. 3. For example, the sorting destination of the item 7A is a sorting container 8C, and the item 7A is outside the movable range RA but within the movable range RB. In this case, the control device 1 formulates an operation plan for the robot arms 51A and 51B so that the item 7A is moved into the movable range RA by pick-and-place by the robot arm 51B, and the moved item 7A is moved to the sorting container 8C by pick-and-place by the robot arm 51A. Similarly, the sorting destination of the item 7B is a sorting container 8A, and the item 7B is outside the movable range RB but within the movable range RA. In this case, the control device 1 moves the item 7B into the movable range RB by pick-and-place by the robot arm 51A, and the moved item 7B is moved to the sorting container 8A by pick-and-place by the robot arm 51B.

[0038] In addition, when judging whether or not a transfer is required, the control device 1 may judge whether or not the item 7 is within the movable range of the robot arm based on the position of the item 7 at the time when the item 7 is carried by the first conveyance machine 52A to the position closest to the robot arm (the front position in Fig. 3). For example, the item 7D is outside the movable ranges RA and RB at the time shown in Fig. 3, but is within the movable ranges RA and RB when carried to the front position of the robot arms 51A and 51B, so that the control device 1 considers the item 7D to be within the movable ranges RA and RB when judging whether or not a transfer is required.

[0039] The robot arm 51C picks and places the item 7 on the third conveying machine 52C that has been subjected to exceptional processing such as the affixing of a barcode and / or inspection processing, to a sorting container 8 placed on the sorting container table 55.

[0040] (4) Functional Blocks FIG. 4 is an example of a functional block showing an overview of the processing of the control device 1. In summary, the control device 1 performs an operation plan for the robot 51 to sort the items 7 into the appropriate sorting containers 8, and updates (re-determines) the operation plan when the sorting containers 8 are replenished. As a result, even if the arrangement of the sorting containers 8 is changed, the control device 1 operates the robot 51 to sort the items 7 into the appropriate sorting containers 8. The processor 11 of the control device 1 functionally has a robot control unit 14, an operation planning unit 15, a determination unit 16, and a sorting container control unit 17. Note that in FIG. 4, the blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to FIG. 4. The same applies to other functional block diagrams described later.

[0041] The robot control unit 14 controls the robot 51 to operate according to the operation plan determined by the operation planning unit 15 by transmitting the control signal S2 to the robot 51. Specifically, the robot control unit 14 executes control (e.g., position control or torque control of the joints of the robot 51) for the robot 51 to execute each subtask constituting the operation sequence generated by the operation planning unit 15 according to the operation plan at a respective determined execution timing (time step). In addition, when an updated operation sequence is supplied from the operation planning unit 15 during control of the robot 51 based on the operation sequence, the robot control unit 14 controls the operation of the robot 51 based on the updated operation sequence.

[0042] Note that the function equivalent to the robot control unit 14 may be provided in the robot 51 instead of the control device 1. In this case, the robot 51 receives a control signal S2 representing a movement sequence generated by the movement planning unit 15, and executes joint position control or torque control for realizing the movement sequence.

[0043] The motion planning unit 15 generates a motion sequence for the robot 51 related to pick-and-place for moving the objects 7 to be sorted to the sorting containers 8 according to the classification of the objects 7 based on the application information IA, the sorting information IB, and the measurement signal S1. Then, the motion planning unit 15 supplies the generated motion sequence to the robot control unit 14. Here, the motion sequence includes information indicating the execution order and execution timing of each subtask.

[0044] In addition, when the operation planning unit 15 receives a judgment result from the judgment unit 16 indicating that the sorting container 8 has been replenished, it updates (re-determines) the operation plan for picking and placing the items 7 to be sorted that have not yet been sorted into the sorting container 8, and generates an operation sequence for the robot 51.

[0045] Here, a supplementary explanation is given regarding the necessity of updating the above-mentioned motion plan. When the sorting containers 8 are replenished, the options in the motion plan of the robot 51 increase, in that the replenished empty sorting containers 8 are set as the sorting destination of the items 7, and the placement of the existing sorting containers 8 may also shift (slide) as the sorting containers 8 are replenished. Taking the above into consideration, when the sorting containers 8 are replenished, the motion plan unit 15 updates (re-determines) the motion plan regarding the pick-and-place of the items 7 to be sorted that have not yet been sorted into the sorting containers 8. This allows the motion plan unit 15 to suitably formulate the motion plan of the robot 51 so as to move the items 7 to the sorting containers 8 that are the appropriate sorting destinations.

[0046] The determination unit 16 determines whether the sorting containers 8 have been replenished and supplies the determination result to the operation planning unit 15. In this case, the determination unit 16 may determine whether the sorting containers 8 have been replenished based on a measurement signal S1 supplied from the measuring device 4, or may determine whether the sorting containers 8 have been replenished based on a signal supplied from the sorting container control unit 17. In the former case, the determination unit 16 determines that the sorting containers 8 have been replenished when it detects that a new sorting container 8 has been placed in a predetermined location (the sorting container table 54 in FIG. 3) based on an image captured by the camera 41. In the latter case, the determination unit 16 determines that the sorting containers 8 have been replenished when it receives a signal from the sorting container control unit 17 notifying the completion of the execution of the process of replenishing the sorting containers 8.

[0047] In the example of FIG. 3, the determination unit 16 may only determine whether the sorting containers 8 on the sorting container table 54 have been replenished, and may not determine whether the sorting containers 8 containing the items 7 that have been subjected to the exception processing have been replenished. In this case, the motion planning unit 15 may not update the motion plans for the robot arms 51A and 51B even if the sorting containers 8 have been replenished on the sorting container table 55. In this case, the motion planning unit 15 executes, for example, the motion plans for the robot arms 51A and 51B and the motion plan for the robot arm 51C independently. Unless otherwise specified, the determination and update of the motion plans described below are intended to be directed to the motion plans for the robot arms 51A and 51B, and do not take into account the state of the exception processing line (robot 51C, third conveying machine 52C, sorting container table 55).

[0048] The sorting container control unit 17 controls the removal and refilling of the sorting containers 8. In the example of Fig. 3, when a predetermined condition is satisfied, the sorting container control unit 17 controls the sorting container refilling mechanism 53 and the sorting container table 54 based on the control signal S2 to remove the sorting container 8 on the sorting container table 54 to the second conveying machine 52B and refill the sorting container table 54 with a new sorting container 8.

[0049] In a first example of the removal and refilling of the sorting containers 8, when there is a sorting container 8 that has become unable to accommodate any more items 7 (also referred to as the "reference state"), the sorting container control unit 17 recognizes the sorting container 8 as a target for removal, and performs control to remove the sorting container 8 to the second conveying machine 52B and refill a new sorting container 8. In this case, the sorting container control unit 17 may determine whether the sorting container 8 has reached the reference state based on the measurement signal S1 of the measuring device 4, or may determine based on the progress of the operation plan formulated by the robot control unit 14. In the former determination method, the sorting container control unit 17 recognizes, for example, the type and number of items 7 accommodated in the target sorting container 8 based on the measurement signal S1, and determines whether the upper limit number of items 7 corresponding to the recognized type has been reached. In this case, for example, information regarding the upper limit number of items 7 that can be accommodated in the sorting container 8 for each type of item 7 described above is pre-stored in the memory 12 or the like. In the latter judgment method, the sorting container control unit 17 estimates the type and number of items 7 placed in the sorting container 8 at the current time based on the progress of the operation plan formulated by the operation planning unit 15, and determines whether or not the state is in the reference state by comparing the estimated result with the information on the maximum storage capacity mentioned above.

[0050] In a second example of the removal and refilling of the sorting containers 8, the sorting container control unit 17 determines that there are no candidates for the sorting destination of the item 7 when the item 7 to be sorted belongs to a new classification that cannot be sorted into any of the sorting containers 8A-8D placed on the sorting container table 54. In this case, for example, based on the identification information of the item 7 to be sorted supplied from the BCR 42 and the sorting information IB, the sorting container control unit 17 determines that there are no candidates for the sorting destination when the item 7 belongs to a different classification from the items 7 contained in the sorting containers 8A-8D (and none of the sorting containers 8A-8D are empty). Then, the sorting container control unit 17 controls the removal of one of the sorting containers 8A-8D to the second conveying machine 52B and the refilling of the sorting container refilling mechanism 53 with a new sorting container 8. In this case, the sorting container control unit 17 selects, for example, the sorting container 8 that has the highest storage rate of items 7 (the ratio of the actual storage number to the maximum storage number) or that contains the largest number of items 7 among the sorting containers 8A-8D as the target for transport to the second conveying machine 52B. This allows the sorting container control unit 17 to suitably control the transport and replenishment of the sorting containers 8 so that the sorting of the items 7 is performed appropriately while minimizing the number of sorting containers 8 used.

[0051] The above-mentioned process of determining whether to remove or replenish the sorting containers 8 may be executed when the operation planning unit 15 sets a target task. In this case, the sorting container control unit 17 executes the removal or replenishment of the sorting containers 8 in response to a request for removal or replenishment of the sorting containers 8 from the operation planning unit 15. The determination of whether removal or replenishment is necessary when setting a target task will be described later with reference to FIG. 6.

[0052] Here, each of the components of the robot control unit 14, the operation planning unit 15, the determination unit 16, and the sorting container control unit 17 can be realized, for example, by the processor 11 executing a program. Also, each component may be realized by storing a necessary program in an arbitrary non-volatile storage medium and installing it as necessary. At least a part of these components may be realized by any combination of hardware, firmware, and software, without being limited to being realized by software by a program. Also, at least a part of these components may be realized by using an integrated circuit that is programmable by a user, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the program composed of each of the above components may be realized using this integrated circuit. Also, at least a part of each component may be composed of an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum computer control chip. In this way, each component may be realized by various hardware. The above is the same in other embodiments described later. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0053] (5) Motion Planning Details Next, a detailed description will be given of the motion plan of the robot 51 executed by the motion planning unit 15. As an example, a method for generating a motion sequence, which is a sequence of subtasks for each time step of the robot 51, based on a goal state expressed by temporal logic will be described below.

[0054] (5-1) Application Information First, the data structure of application information IA used in the operation plan by the operation planner 15 will be described.

[0055] Fig. 5 shows an example of the data structure of application information IA. As shown in Fig. 5, application information IA includes abstract state designation information I1, constraint condition information I2, operation limit information I3, subtask information I4, abstract model information I5, and object model information I6.

[0056] The abstract state designation information I1 is information that designates an abstract state that needs to be defined when generating an action sequence. This abstract state is an abstract state of an object in the working space, and is defined as a proposition to be used in the target logical formula described later.

[0057] The constraint information I2 is information indicating constraint conditions when executing a target task. For example, when the target task is pick-and-place, the constraint information I2 indicates a constraint condition that the robot 51 (robot arm) must not come into contact with an obstacle, a constraint condition that the robots 51 (robot arms) must not come into contact with each other, and the like.

[0058] The motion limit information I3 indicates information on the motion limit of the robot 51 controlled by the control device 1. The motion limit information I3 is, for example, information that specifies the upper limit of the speed, acceleration, or angular velocity of the robot 51. Note that the motion limit information I3 may be information that specifies the motion limit for each movable part or joint of the robot 51.

[0059] The subtask information I4 indicates information on subtasks that can be accepted by the robot 51. For example, when the target task is pick-and-place, the subtask information I4 specifies, as subtasks, reaching, which is the movement of the robot arm of the robot 51, and grasping, which is the gripping by the robot arm.

[0060] The abstract model information I5 is information on an abstract model that abstracts dynamics in the working space. For example, the abstract model is represented by a model that abstracts real-world dynamics using a hybrid system, as described below. The abstract model information I5 includes information indicating a condition for switching dynamics in the hybrid system described above. For example, in the case of pick-and-place, in which the robot 51 grasps an object and moves it to a predetermined position, an example of the switching condition is that the object cannot be moved unless it is grasped by the robot 51.

[0061] The object model information I6 is information on the object model of each object in the working space to be recognized from the measurement signal S1 generated by the measuring device 4. The above-mentioned objects include, for example, the robot 51, obstacles, tools handled by the robot 51 and other objects, and working bodies other than the robot 51. The object model information I6 includes, for example, information required for the control device 1 to recognize the type, position, posture, and currently executed operation of each of the above-mentioned objects, and three-dimensional shape information such as CAD (Computer Aided Design) data for recognizing the three-dimensional shape of each object. The former information includes parameters of an inference device obtained by learning a learning model in machine learning such as a neural network. For example, when an image is input, this inference device is trained in advance to output the type, position, posture, and the like of an object that is a subject in the image.

[0062] In addition to the above-mentioned information, the application information IA may store various information related to the generation process of the operation sequence.

[0063] (5-2) Motion planning function block 6 is an example of a functional block diagram showing an overview of the processing of the motion planning unit 15. The motion planning unit 15 functionally includes a target task setting unit 70, an abstract state setting unit 71, a target logical formula generating unit 72, a time step logical formula generating unit 73, an abstract model generating unit 74, a control input generating unit 75, and a motion sequence generating unit 76.

[0064] The target task setting unit 70 sets a target task based on the measurement signal S1 and the sorting information IB supplied from the item 7. In this case, the target task setting unit 70 determines the item 7 to be sorted and the sorting container 8 into which the item 7 is to be sorted.

[0065] The abstract state setting unit 71 sets an abstract state in the work space based on the measurement signal S1, abstract state designation information I1, and object model information I6 supplied from the measurement device 4. In this case, the abstract state setting unit 71 recognizes objects (items 7, sorting containers 8, etc.) in the work space that need to be considered in the work of the robot 51, and generates a recognition result "Im" for the object. Then, based on the recognition result Im, the abstract state setting unit 71 defines a proposition for expressing each abstract state that needs to be considered in the work of the robot 51 as a logical formula. The abstract state setting unit 71 supplies information representing the set abstract state (also referred to as "abstract state setting information IS") to the target logical formula generation unit 72.

[0066] Based on the abstract state setting information IS, the target logical formula generation unit 72 converts the target task into a logical formula of temporal logic (also called "target logical formula Ltag") that represents a final achieved state. In this case, the target logical formula generation unit 72 adds a constraint condition that should be satisfied in the execution of the target task to the target logical formula Ltag by referring to the constraint condition information I2 of the application information IA. Then, the target logical formula generation unit 72 supplies the generated target logical formula Ltag to the time-step logical formula generation unit 73.

[0067] The time-step logical formula generation unit 73 converts the target logical formula Ltag supplied from the target logical formula generation unit 72 into a logical formula (also called a "time-step logical formula Lts") that represents a state at each time step. Then, the time-step logical formula generation unit 73 supplies the generated time-step logical formula Lts to the control input generation unit 75.

[0068] The abstract model generation unit 74 generates an abstract model "Σ" that abstracts the real dynamics in the working space, based on the abstract model information I5 of the application information IA and the recognition result Im supplied from the abstract state setting unit 71. In this case, the abstract model generation unit 74 regards the target dynamics as a hybrid system in which continuous dynamics and discrete dynamics are mixed, and generates an abstract model Σ based on the hybrid system. The method of generating the abstract model Σ will be described later. The abstract model generation unit 74 supplies the generated abstract model Σ to the control input generation unit 75.

[0069] The control input generation unit 75 determines a control input to the robot 51 for each time step that satisfies the time step logical formula Lts supplied from the time step logical formula generation unit 73 and the abstract model Σ supplied from the abstract model generation unit 74 and optimizes an evaluation function (for example, a function representing the amount of energy consumed by the robot). Then, the control input generation unit 75 supplies information indicating the control input to the robot 51 for each time step (also referred to as "control input information Icn") to the operation sequence generation unit 76.

[0070] The movement sequence generating unit 76 generates a movement sequence, which is a sequence of subtasks, based on the control input information Icn supplied from the control input generating unit 75 and the subtask information I4 of the application information IA, and supplies the movement sequence to the robot 51.

[0071] (5-3) Target task setting section The target task setting unit 70 determines the items 7 to be sorted and the sorting container 8 into which the items 7 will be sorted, and sets a target task of moving the items 7 to the destination sorting container 8 by pick-and-place.

[0072] Here, a method for determining the item 7 to be sorted and the sorting container 8 to be sorted will be described. First, the destination task setting unit 70, for example, identifies the item 7 that has passed through the BCR 42 as the item 7 to be sorted, and performs sorting classification (e.g., classification by type or lot) of the item 7 based on the reading result of the BCR 42 for the identified item 7 and the sorting information IB. For example, when the sorting information IB indicates a correspondence relationship between the ID of the item 7 and the sorting classification, the destination task setting unit 70 recognizes the sorting classification of the item 7 based on the ID of the item 7 read by the BCR 42 and the above-mentioned correspondence. Note that, when the information read by the BCR 42 indicates the sorting classification of the item 7, the destination task setting unit 70 recognizes the sorting classification of the item 7 to be sorted based on the reading result of the BCR 42 without using the sorting information IB.

[0073] Next, specific examples of a method for determining a destination sorting container 8 based on the classification of the item 7 to be sorted will be described for different cases. For example, when an item 7 having the same sorting destination (i.e., the same classification) as the item 7 to be sorted is already contained in one of the sorting containers 8, the target task setting unit 70 sets the sorting destination of the item 7 to be sorted to that sorting container 8. Here, the target task setting unit 70 specifies the classification of the item 7 already contained in the sorting container 8 based on, for example, the recognition result of the classification of the item 7 when a past operation plan was formulated. Note that a response when the sorting container 8 specified as the sorting destination is in the reference state will be described later.

[0074] In addition, if there are no other items 7 that have the same sorting destination as the item 7 to be sorted contained in any of the sorting containers 8 and there is an empty sorting container 8, the target task setting unit 70 determines that the empty sorting container 8 is the sorting destination for the item 7 to be sorted.

[0075] Furthermore, when the destination task setting unit 70 determines that none of the sorting containers 8 contain an item 7 that has the same sorting destination as the item 7 to be sorted, and that there are no empty sorting containers 8, it determines that one of the sorting containers 8 needs to be removed and a new sorting container 8 needs to be refilled. Therefore, in this case, the destination task setting unit 70 supplies a request signal for removal and refilling of the sorting container 8 to the sorting container control unit 17. Then, when the destination task setting unit 70 detects that an empty sorting container 8 has been refilled based on the measurement signal S1 or a refill completion notification from the sorting container refill mechanism 53, it determines the refilled empty sorting container 8 as the sorting destination for the item 7 to be sorted.

[0076] In addition, when an item 7 that is to be sorted to the same sorting destination as the item 7 to be sorted is already contained in one of the sorting containers 8 and the sorting container 8 is in a reference state, the destination task setting unit 70 determines that the sorting container 8 needs to be removed and refilled. Therefore, in this case, the destination task setting unit 70 supplies a request signal for removing the sorting container 8 in the reference state and refilling a new sorting container 8 to the sorting container control unit 17. Then, when the destination task setting unit 70 detects that an empty sorting container 8 has been refilled based on the measurement signal S1 or a refill completion notification from the sorting container refill mechanism 53, the destination task setting unit 70 determines the refilled empty sorting container 8 as the sorting destination of the item 7 to be sorted. Note that, when an empty sorting container 8 is present, the destination task setting unit 70 may set the destination task with the empty sorting container 8 as the sorting destination without waiting for the above-mentioned removal and refilling, and proceed with the formulation of the operation plan.

[0077] When identification information (ID) is assigned to the sorting containers 8 and the classification (type or lot) of the items 7 to be contained in each sorting container 8 is specified in the sorting information IB, the target task setting unit 70 recognizes the sorting destination sorting container 8 based on the read information of the barcode read from the items 7 to be sorted and the sorting information IB. In this case, each sorting container 8 is provided with an AR marker or the like, so that its unique identification information (ID) can be identified based on measurements by the camera 41 or the like.

[0078] Then, the target task setting unit 70 supplies the result of determining the items 7 to be sorted and the sorting containers 8 into which the items 7 are to be sorted to the target logical formula generating unit 72 and the like as information on the target task. As a result, an operation plan is formulated for each set target task, and an operation sequence of the robot 51 for the items 7 to be sorted is sequentially generated. Note that, instead of setting a target task for each item 7 that has passed through the BCR 42, the target task setting unit 70 may set a target task for each item 7 that has passed through the BCR 42 in a predetermined period (for example, a period determined every predetermined number of seconds). In this case, the control device 1 may stop the transportation by the first conveying machine 52A when the operation plan is being formulated and when the robot 51 is operating. In this case, the control device 1 repeatedly executes a cycle including, for example, the formulation of an operation plan, the operation of the robot 51 based on the operation plan, and transportation for a predetermined period by the first conveying machine 52A.

[0079] (5-4) Abstract State Setting Part First, the abstract state setting unit 71 refers to the object model information I6 and generates a recognition result Im by analyzing the measurement signal S1, which is an output signal of an external sensor such as the camera 41, using a technology for recognizing the environment of the work space. The technology for recognizing the environment of the work space described above may be any of image processing technology, image recognition technology, voice recognition technology, and technology using RFID (Radio Frequency Identifier). The recognition result Im includes information such as the type, position, and posture of objects (including the object 7 to be sorted) in the work space. The objects in the work space are, for example, the robot 51, the object 7, obstacles, and other work bodies (people or other objects performing work other than the robot 51). Note that the abstract state setting unit 71 recognizes, for example, the object 7 that exists in a position where the BCR 42 can read the barcode at the time when the BCR 42 reads the barcode of the object 7 as the object 7 to be sorted.

[0080] Next, the abstract state setting unit 71 sets an abstract state in the work space based on the recognition result Im and the abstract state designation information I1 of the application information IA. In this case, the abstract state setting unit 71 first refers to the abstract state designation information I1 and recognizes the abstract state to be set in the work space.

[0081] FIG. 7 shows an overhead view of a pick-and-place workspace that simplifies the bulk sorting work shown in FIG. 3. In the workspace shown in FIG. 7, there are two robot arms 51a, 51b, four articles 7 (7a-7d) placed on a conveying machine 52a, an obstacle 9, and a sorting container 8a that is the destination of the articles 7. Thus, in FIG. 7, there is only one sorting container 8 for the sake of simplicity. Hereinafter, the internal area of ​​the sorting container 8a where the articles 7 should be placed is also referred to as "area G". The robot arms 51a, 51b are examples of a "first robot" and a "second robot", respectively.

[0082] In this case, first, the abstract state setting unit 71 recognizes the state of the article 7, the range in which the obstacle 9 exists, the state of the robot 51, the range in which the area G exists, and the like.

[0083] Here, the abstract state setting unit 71 recognizes the position vectors "x1" to "x4" of the centers of the articles 7a to 7d as the positions of the articles 7a to 7d. In addition, the abstract state setting unit 71 recognizes the position vector "x r1 " and the position vector "x r2" are recognized as the positions of the robot arm 51a and the robot arm 51b. When the articles 7a to 7d are being transported by the transport machine 52a at a predetermined transport speed, the abstract state setting unit 71 may express the position vectors x1 to x4 by a predetermined model (function) having as parameters the initial position indicated by the recognition result Im, the transport speed of the transport machine 52a, and the length of time elapsed from the initial position. Information on the above-mentioned model (function) is included in advance in, for example, the application information IA. When the control device 1 stops the transport by the transport machine 52a during the operation planning and the operation of the robot 51, the abstract state setting unit 71 sets the position vectors x1 to x4 to fixed values ​​based on the recognition result Im.

[0084] Similarly, the abstract state setting unit 71 recognizes the postures of the articles 7a to 7d, the existence range of the obstacle 9, the existence range of the area G, etc. Note that, for example, when the abstract state setting unit 71 regards the obstacle 9 as a rectangular parallelepiped and the area G as a rectangle, it recognizes the position vectors of each vertex of the obstacle 9 and the area G.

[0085] The abstract state setting unit 71 also determines the abstract state to be defined in the target task by referring to the abstract state designation information I1. In this case, the abstract state setting unit 71 determines a proposition indicating an abstract state based on the recognition result Im (e.g., the number of each type of object) regarding the objects existing in the work space and the abstract state designation information I1.

[0086] In the example of FIG. 7, the abstract state setting unit 71 assigns identification labels "1" to "4" to the items 7a to 7d identified by the recognition result Im, respectively. In addition, the abstract state setting unit 71 assigns the proposition "g i In addition, the abstract state setting unit 71 assigns an identification label "O" to the obstacle 9 and defines a proposition "o i Furthermore, the abstract state setting unit 71 defines a proposition "h" that the robot arms 51a and 51b interfere with each other.

[0087] In this way, the abstract state setting unit 71 recognizes the abstract state to be defined by referring to the abstract state designation information I1, and determines the proposition expressing the abstract state (in the above example, g i ,o i , h, etc.) are defined according to the number of articles 7, the number of robots 51, the number of obstacles 9, etc. Then, the abstract state setting unit 71 supplies information indicating a proposition expressing an abstract state to the target logical formula generation unit 72 as abstract state setting information IS.

[0088] (5-5) Goal formula generator First, the target logical expression generating unit 72 converts the target task set by the target task setting unit 70 into a logical expression using temporal logic.

[0089] For example, in the example of FIG. 7, assume that a goal task of "Eventually, the item (i=2) is present in the area G" is given. In this case, the goal logical formula generating unit 72 generates the goal task by dividing the goal task into an operator "◇" corresponding to "eventually" in the linear logical formula (LTL: Linear Temporal Logic) and a proposition "g i " to generate the logical formula "◇g2". Furthermore, the target logical formula generation unit 72 may express a logical formula using any temporal logic operator other than the operator "◇" (logical product "∧", logical sum "∨", negation "￢", logical inclusion "⇒", always "□", next "○", until "U", etc.). Furthermore, the logical formula may be expressed using any temporal logic, such as MTL (Metric Temporal Logic) or STL (Signal Temporal Logic), not limited to linear temporal logic.

[0090] Next, the target logical expression generating unit 72 generates a target logical expression Ltag by adding the constraint condition indicated by the constraint condition information I2 to the logical expression indicating the objective task.

[0091] For example, when the constraint condition information I2 includes two constraint conditions corresponding to the pick-and-place shown in FIG. 7, namely, "the robot arms do not interfere with each other" and "the object i does not interfere with the obstacle O", the target logical formula generating unit 72 converts these constraint conditions into a logical formula. Specifically, the target logical formula generating unit 72 converts the proposition "o i Using " and proposition "h", the two constraints above are converted into the following logical formulas, respectively. □￢h ∧ i □￢o i

[0092] Therefore, in this case, the target logical formula generation unit 72 generates the following target logical formula Ltag by adding the logical formulas of these constraint conditions to the logical formula "◇g2" corresponding to the objective task "finally, the item (i=2) is present in area G." (◇g2)∧(□￢h)∧(∧ i □￢o i )

[0093] In reality, the constraint conditions corresponding to pick-and-place are not limited to the above two, and there are other constraint conditions such as "the robot arm does not interfere with the obstacle O," "multiple robot arms do not pick up the same item," and "items do not come into contact with each other." These constraint conditions are also stored in the constraint condition information I2 and reflected in the target logical formula Ltag.

[0094] (5-6) Time step formula generator The time-step logical formula generation unit 73 determines the number of time steps (also called the "target number of time steps") to complete the target task, and determines a combination of propositions that express the state at each time step that satisfies the target logical formula Ltag in the target number of time steps. Since there are usually multiple combinations, the time-step logical formula generation unit 73 generates a logical formula that combines these combinations with a logical sum as the time-step logical formula Lts. The above combinations become candidates for a logical formula that expresses a sequence of operations to be commanded to the robot 51, and are hereinafter also called "candidates φ."

[0095] Here, a specific example of the process of the time step logical expression generating unit 73 when a goal task of "finally, the item (i=2) exists in the area G" is set as exemplified in the explanation of FIG. 7 will be described.

[0096] In this case, the following target logical formula Ltag is supplied from the target logical formula generator 72 to the time step logical formula generator 73. (◇g2)∧(□￢h)∧(∧ i □￢o i ) In this case, the time step formula generator 73 generates the proposition "g i " is extended to include the concept of time steps, i,k " is used. Here, the proposition "g i,k " is a proposition that "item i exists in area G at time step k." If the target number of time steps is set to "3," the target logical formula Ltag can be rewritten as follows. (◇g 2,3 )∧(∧ k=1,2,3 □￢h k )∧(∧ i,k=1,2,3 □￢o i,k )

[0097] Also, ◇g 2,3 can be rewritten as shown in the following equation:

[0098]

number

[0099] At this time, the above-mentioned target logical formula Ltag is expressed by the logical sum (φ1∨φ2∨φ3∨φ4) of the four candidates “φ1” to “φ4” shown below.

[0100]

number

[0101] Therefore, the time-step logical expression generating unit 73 determines the logical sum of the four candidates φ1 to φ4 as the time-step logical expression Lts. In this case, the time-step logical expression Lts is true when at least one of the four candidates φ1 to φ4 is true.

[0102] Next, a method for setting the target number of time steps will be explained in more detail.

[0103] The time step logical formula generating unit 73 determines the target number of time steps based on, for example, the expected time of the operation stored in advance in the memory 12 etc. In this case, the time step logical formula generating unit 73 calculates the target number of time steps from the expected time based on information on the time width per time step stored in advance in the memory 12 etc. In another example, when information indicating the target number of time steps to be set is stored in advance in the memory 12 etc., the time step logical formula generating unit 73 determines the target number of time steps by referring to the information.

[0104] Preferably, the time-step logical formula generating unit 73 sets the target number of time steps to a predetermined initial value. Then, the time-step logical formula generating unit 73 gradually increases the target number of time steps until a time-step logical formula Lts that allows the control input generating unit 75 to determine the control input is generated. In this case, if the control input generating unit 75 performs optimization processing using the set target number of time steps and is unable to derive an optimal solution, the time-step logical formula generating unit 73 adds a predetermined number (an integer equal to or greater than 1) to the target number of time steps.

[0105] (5-7) Abstract Model Generation The abstract model generation unit 74 generates an abstract model Σ based on the abstract model information I5 and the recognition result Im. Here, the abstract model information I5 records information necessary for generating the abstract model Σ. For example, a generic abstract model that does not specify the position and number of items, the position of the area where the items are placed, the number of robots 51, etc. is recorded in the abstract model information I5. Then, the abstract model generation unit 74 generates the abstract model Σ by reflecting the recognition result Im to the generic abstract model recorded in the abstract model information I5, which includes the dynamics of the robot 51. As a result, the abstract model Σ becomes a model that abstractly represents the state of the object in the working space and the dynamics of the robot 51. The state of the object in the working space indicates the position and number of items, the position and number of the area (sorting container 8) where the items are placed, the number of robots 51, etc.

[0106] When there is another work object other than the robot 51 to be controlled, information on the abstracted dynamics of the other work object may be included in the abstract model information I5. In this case, the abstract model Σ is a model in which the state of the object in the working space, the dynamics of the robot 51, and the dynamics of the other work object are abstractly represented.

[0107] Here, when the robot 51 is working on a target task, the dynamics in the workspace frequently switches. For example, when the robot arm is holding an item i, the item i can be moved, but when the robot arm is not holding the item i, the item i cannot be moved.

[0108] In consideration of the above, in this embodiment, the action of grabbing the item i is expressed as a logical variable “δ i In this case, for example, the abstract model generation unit 74 can determine the abstract model Σ to be set for the work space shown in FIG.

[0109]

number

[0110] Here, "u j " indicates a control input for controlling the robot hand j ("j=1" is the robot hand of the robot arm 51a, and "j=2" is the robot hand of the robot arm 51b), "I" indicates a unit matrix, and "0" indicates a zero-row matrix. Note that the control input is assumed to be velocity here as an example, but it may also be acceleration. Also, "δ j,i " is a logical variable that is "1" when the robot hand j is holding the item i, and is "0" otherwise. Also, "x r1 ", "x r2 " indicates the position vector of robot hand j (j = 1, 2), "x1" to "x4" indicate the position vector of item i (i = 1 to 4). "h(x)" is a variable that satisfies "h(x) ≧ 0" when a robot hand is close enough to an item to grasp it, and satisfies the following relationship with logical variable δ. δ=1 ⇔ h(x)≧0 In this equation, if the robot hand is close enough to an object to grasp it, it is assumed that the robot hand is grasping the object, and the logical variable δ is set to 1.

[0111] Here, equation (1) is a differential equation that shows the relationship between the state of the object at time step k and the state of the object at time step k + 1. In the above equation (1), the grip state is represented by a logical variable that is a discrete value, and the movement of the object is represented by a continuous value, so equation (1) shows a hybrid system.

[0112] Equation (1) takes into consideration only the dynamics of the robot hand, which is the end effector of the robot 51 that actually grasps an object, rather than the detailed dynamics of the entire robot 51. This makes it possible to suitably reduce the amount of calculation in the optimization process performed by the control input generation unit 75.

[0113] Furthermore, the abstract model information I5 records logical variables corresponding to the action of switching dynamics (the action of grasping the item i in the case of pick-and-place) and information for deriving the difference equation of formula (1) from the recognition result Im. Therefore, the abstract model generation unit 74 can determine an abstract model Σ that is suited to the environment of the target workspace based on the abstract model information I5 and the recognition result Im, even if the positions and number of items, the positions and number of areas (sorting containers 8) in which the items are placed, the number of robots 51, etc. vary.

[0114] Note that the abstract model generating unit 74 may generate a model of a mixed logical dynamical (MLD) system or a hybrid system that combines a Petri net, an automaton, and the like, instead of the model shown in equation (1).

[0115] (5-8) Control Input Generation Unit The control input generation unit 75 determines an optimal control input for the robot 51 for each time step based on the time step logical formula Lts supplied from the time step logical formula generation unit 73 and the abstract model Σ supplied from the abstract model generation unit 74. In this case, the control input generation unit 75 defines an evaluation function for the target task, and solves an optimization problem of minimizing the evaluation function with the abstract model Σ and the time step logical formula Lts as constraints. The evaluation function is stored in advance in the memory 12, for example.

[0116] For example, when a pick-and-place task is set as a target task, the control input generating unit 75 calculates the distance "d k " and the control input "u k The evaluation function is determined so that the above-mentioned distance d k In the case of a target task in which "item (i=2) is ultimately present in region G," corresponds to the distance between item (i=2) and region G at time step k.

[0117] In this case, the control input generator 75 calculates the distance d k The square of the norm of and the control input u k The control input generation unit 75 determines the sum of the square of the norm of the abstract model Σ and the time step logical formula Lts (i.e., the candidate φ i The constrained mixed integer optimization problem shown in the following equation (2) is solved, with the constraint being the logical sum of

[0118]

number

[0119] Here, "T" is the number of time steps to be optimized, and may be the target number of time steps, or may be a predetermined number smaller than the target number of time steps, as described later. In this case, preferably, the control input generation unit 75 approximates the logical variables to continuous values ​​(continuous relaxation problem). This allows the control input generation unit 75 to preferably reduce the amount of calculation. Note that when STL is adopted instead of linear logic (LTL), it is possible to describe it as a nonlinear optimization problem. Through such optimization, a control input that specifies the operation of the robot 51, including the transfer of the article 7 between the robots 51 as necessary, is determined.

[0120] Furthermore, when the target number of time steps is long (for example, larger than a predetermined threshold), the control input generation unit 75 may set the number of time steps used for optimization to a value smaller than the target number of time steps (for example, the above-mentioned threshold). In this case, the control input generation unit 75 sequentially obtains the control input u by solving the above-mentioned optimization problem, for example, every time a predetermined number of time steps elapses. k Determine.

[0121] Preferably, the control input generation unit 75 solves the above optimization problem for each predetermined event corresponding to an intermediate state with respect to the achievement state of the target task, and determines the control input u to be used. kmay be determined. In this case, the control input generating unit 75 sets the number of time steps until the next event occurs as the number of time steps used for optimization. The above-mentioned event is, for example, an event in which dynamics in the workspace is switched. For example, when pick-and-place is set as the target task, the event is determined as the robot 51 picking up an item, or the robot 51 finishing transporting one of a plurality of items to be transported to a destination point. The event is, for example, determined in advance, and information specifying the event is stored in advance in the memory 12 or the like.

[0122] (5-9) Operation sequence generator The movement sequence generation unit 76 generates a movement sequence based on the control input information Icn supplied from the control input generation unit 75 and the subtask information I4 of the application information IA. In this case, the movement sequence generation unit 76 recognizes subtasks that the robot 51 can accept by referring to the subtask information I4, and converts the control input for each time step indicated by the control input information Icn into a subtask.

[0123] For example, in the subtask information I4, functions indicating two subtasks, namely, moving (reaching) of the robot hand and grasping (grasping) of the robot hand, are defined as subtasks that the robot 51 can accept when the target task is pick-and-place. In this case, the function "Move" representing reaching is a function that takes as arguments, for example, the initial state of the robot 51 before the execution of the function, the final state of the robot 51 after the execution of the function, and the time required to execute the function. Also, the function "Grasp" representing grasping is a function that takes as arguments, for example, the state of the robot 51 before the execution of the function, the state of the object to be grasped before the execution of the function, and a logical variable δ. Here, the function "Grasp" represents a gripping action when the logical variable δ is "1", and represents a releasing action when the logical variable δ is "0". In this case, the operation sequence generation unit 76 determines the function "Move" based on the trajectory of the robot hand determined by the control input for each time step indicated by the control input information Icn, and determines the function "Grasp" based on the transition of the logical variable δ for each time step indicated by the control input information Icn.

[0124] Then, the motion sequence generating unit 76 generates a motion sequence composed of the function "Move" and the function "Grasp" and supplies the motion sequence to the robot 51. For example, when the target task is "finally, the item (i=2) is present in the area G", the motion sequence generating unit 76 generates a motion sequence of the function "Move", the function "Grasp", the function "Move", and the function "Grasp" for the robot hand closest to the item (i=2). In this case, the robot hand closest to the item (i=2) moves to the position of the item (i=2) by the first function "Move", grasps the item (i=2) by the first function "Grasp", moves to the area G by the second function "Move", and places the item (i=2) in the area G by the second function "Grasp".

[0125] (6) Processing flow FIG. 8 is an example of a flowchart showing an outline of the processing executed by the control device 1 in the first embodiment.

[0126] First, the control device 1 detects the articles 7 to be sorted (step S11). In this case, the control device 1 detects the articles 7 present within a predetermined range as the articles 7 to be sorted, for example, based on the measurement signal S1 generated by the measurement device 4.

[0127] Next, the control device 1 determines the sorting containers 8 to which the articles 7 detected in step S11 are to be sorted (step S12). In this case, the control device 1 determines the sorting containers 8 to which each of the articles 7 detected in step S11 is to be sorted, for example, based on the measurement signal S1 and the sorting information IB. Note that the processing of steps S11 and S12 corresponds to, for example, the processing of the target task setting unit 70 of the operation planning unit 15 described above.

[0128] Next, the control device 1 determines whether or not the sorting containers 8 need to be removed and replenished (step S13). If the control device 1 determines that the sorting containers 8 need to be removed and replenished (step S13; Yes), it performs control to realize the removal and replenishment of the sorting containers 8 (step S14). The processing of steps S13 and S14 corresponds to the processing of the sorting container control unit 17, for example.

[0129] Then, the control device 1 judges whether the sorting containers 8 have been replenished (step S15). The process of step S15 corresponds to, for example, the process of the judgment unit 16. Then, when the control device 1 judges that the sorting containers 8 have been replenished (step S15; Yes), it updates the operation plan based on the arrangement of the sorting containers 8 after replenishment (step S16). In this way, when the sorting containers 8 have been replenished, the control device 1 updates (re-determines) the operation plan for sorting the items 7 based on the latest arrangement of the sorting containers 8. This allows the control device 1 to accurately sort the items 7 based on the latest arrangement of the sorting containers 8. In this case, for example, the control device 1 formulates a new pick-and-place operation plan for the items 7 detected in step S11, and formulates a pick-and-place operation plan for the items 7 for which an operation plan has been prepared and which have not yet been moved to the sorting containers 8 as an updated operation plan. The process of step S16 corresponds to, for example, the process of the operation plan unit 15 other than the target task setting unit 70. On the other hand, if the sorting containers 8 have not been replenished (step S15; No), the control device 1 continues the process of step S14.

[0130] On the other hand, in step S13, if removal and refilling of the sorting containers 8 is not required (step S13; No), the control device 1 determines an operation plan (step S17). The process of step S17 corresponds to the process of the operation planner 15 other than the target task setting unit 70, for example.

[0131] Then, the control device 1 controls the robot 51 based on the operation plan determined in step S16 or step S17 (step S18). Note that the control device 1 may continue to execute the robot control in step S18 until the determined operation plan is completed, and then proceed to the next step. The process of step S18 corresponds to the process of the robot control unit 14, for example.

[0132] Then, the control device 1 determines whether or not the control process of the robot 51, etc. should be terminated (step S19). For example, when the control device 1 detects an external input or the like indicating that the control process of the robot 51, etc. should be stopped, it determines that the control process of the robot 51, etc. should be terminated. When the control device 1 determines that the control process of the robot 51, etc. should be terminated (step S19; Yes), it terminates the processing of the flowchart. On the other hand, when the control device 1 determines that the control process of the robot 51, etc. should not be terminated (step S19; No), it returns the processing to step S11.

[0133] (7) Variations Next, a description will be given of modifications of the first embodiment. The following modifications may be applied in any combination.

[0134] (Variation 1) The block configuration of the motion planning unit 15 shown in FIG. 6 is an example, and various modifications may be made.

[0135] For example, information on candidates φ for a sequence of operations to be commanded to the robot 51 is stored in advance in the memory 12, and the motion planning unit 15 executes an optimization process of the control input generation unit 75 based on the information. As a result, the motion planning unit 15 selects the optimal candidate φ and determines the control input of the robot 51. In this case, the motion planning unit 15 does not need to have functions corresponding to the abstract state setting unit 71, the target logical formula generation unit 72, and the time step logical formula generation unit 73. In this way, information on the execution results of some of the functional blocks of the motion planning unit 15 shown in Fig. 6 may be stored in advance as part of the application information IA.

[0136] In another example, the application information may include design information such as a flowchart for designing an operation sequence corresponding to a target task, and the operation planner 15 may generate the operation sequence by referring to the design information. A specific example of executing a task based on a task sequence designed in advance is disclosed in Japanese Patent Application Laid-Open No. 2017-39170.

[0137] (Variation 2) The work space shown in FIG. 3 is an example, and various modifications may be made to the work space shown in FIG.

[0138] For example, the work space may not be provided with a BCR 42. In this case, for example, the target task setting unit 70 recognizes the type and position of the item 7 on the first conveying machine 52A by applying image recognition technology to the image obtained by the camera 41, and identifies the item 7 present at a predetermined position as the item 7 to be sorted. In addition, the target task setting unit 70 identifies the sorting container 8 to which the item 7 to be sorted is to be sorted, based on the recognized type of item 7. In this case, since the items 7 are sorted according to the type of item 7, which is the image recognition result by the camera 41, there is no need to use the sorting information IB.

[0139] In another example, the article 7 removed from the packing box 6 may be placed directly within the movable range of the robot 51. In this case, the first conveying machine 52A in Fig. 3 may not be present. In yet another example, the sorting container refill mechanism 53 may detect that the sorting container 8 has been removed from the sorting container table 54 or the sorting container table 55 based on the output of its own sensor, and may autonomously perform the refilling work of the sorting container 8 without relying on the control signal S2 from the control device 1.

[0140] In yet another example, the sorting container refilling mechanism 53 may be provided for each of the sorting container tables 54 and 55. In this case, the sorting container refilling mechanism 53 may be provided on the opposite side of the sorting container table 54 or the sorting container table 55 from the position shown in FIG. 3. The sorting container table 54 and the sorting container table 55 may have a function equivalent to the sorting container refilling mechanism 53. In this case, the sorting container table 54 and the sorting container table 55 may have a mechanism for sending a refilling container 8 from inside to the position where the sorting container 8 was located when any of the sorting containers 8 is transported to the second conveying machine 52B. In addition, instead of the sorting container refilling mechanism 53, the robot arms 51A to 51C may perform the task of refilling the sorting containers 8.

[0141] Furthermore, the control process of the robot 51 and the like shown in FIG. 4 and FIG. 8 can be suitably applied not only to sorting incoming goods in a warehouse but also to receiving operations in the assembly manufacturing industry, food manufacturing industry, or logistics industry.

[0142] (Variation 3) The sorting control system 100 includes a warehouse management system (WMS), and the control device 1 may cooperate with the warehouse management system to execute processes related to bulk sorting.

[0143] FIG. 9 is a schematic diagram of a sorting control system 100A in the third modification. As shown in FIG. 9, the sorting control system 100A includes a warehouse management device 2 that manages information in a warehouse management system. The control device 1 and the warehouse management device 2 cooperate to execute the process executed by the control device 1 shown in FIG. 1. For example, the warehouse management device 2 manages the barcode reading result by the BCR 42 and the sorting information IB, and the control device 1 detects an item 7 that has passed through the BCR 42 by a camera 41 or the like, and receives information on a sorting container 8 to which the item 7 is to be sorted from the warehouse management device 2. In this way, according to this modification, the control device 1 can appropriately determine a sorting container 8 to which the item 7 is to be sorted by performing data communication with the warehouse management device 2, and can formulate an operation plan for the robot 51 based on the above-mentioned embodiment.

[0144] <Second embodiment> Fig. 10 is a schematic configuration diagram of a control device 1X in the second embodiment. As shown in Fig. 10, the control device 1X mainly includes an operation planning means 15X and a determination means 16X. The control device 1X may be composed of a plurality of devices.

[0145] The motion planning means 15X determines a motion plan for the robot regarding the task of sorting a plurality of objects into a plurality of sorting containers. Here, "sorting a plurality of objects into a plurality of sorting containers" refers to moving a plurality of objects into the sorting containers so that the objects are in a state in which they are sorted according to the classification of each object. The motion planning means 15X can be, for example, the motion planning unit 15 in the first embodiment.

[0146] The determination means 16X determines whether the sorting containers have been replenished. Here, "replenishment of sorting containers" refers to the installation of new empty sorting containers that are candidates for sorting destinations of articles. The determination means 16X can be, for example, the determination unit 16 in the first embodiment.

[0147] Then, when the sorting containers are replenished, the motion planning means 15X updates the motion plan based on the arrangement of the sorting containers after replenishment. Here, "updating the motion plan" refers to formulating a new motion plan based on the arrangement of the sorting containers after replenishment, instead of the motion plan formulated based on the state of the sorting containers before replenishment. In this case, the motion plan is formulated for items for which sorting work has not been performed.

[0148] FIG. 11 is an example of a flowchart executed by the control device 1X in the second embodiment. First, the motion planning means 15X determines a motion plan for the robot regarding the task of sorting a plurality of objects into a plurality of sorting containers (step S21). Then, the determination means 16X determines whether the sorting containers have been replenished (step S22). Then, if the sorting containers have been replenished (step S22; Yes), the motion plan is updated based on the arrangement of the sorting containers after replenishment (step S23). Note that, if the sorting containers have not been replenished (step S22; No), the determination means 16X continues to determine whether the sorting containers have been replenished in step S22. Note that, if the motion plan is executed normally, etc., the control device 1X may end the processing of this flowchart without executing step S23.

[0149] According to the configuration of the second embodiment, the control device 1X can formulate an appropriate operation plan even when the sorting containers are replenished, and can cause the robot to suitably perform the work of sorting the articles.

[0150] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer readable media and can be supplied to a computer, such as a processor. The non-transitory computer readable medium includes various types of tangible storage media. Examples of the non-transitory computer readable medium include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory)). The program may also be supplied to the computer by various types of transitory computer readable media. Examples of the transitory computer readable medium include electric signals, optical signals, and electromagnetic waves. The transitory computer readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

[0151] In addition, some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0152] [Appendix 1] a motion planning means for determining a motion plan for the robot with respect to a task of sorting a plurality of articles into a plurality of sorting containers; A determination means for determining whether the sorting container has been replenished, The operation planning means, when the sorting containers are replenished, updates the operation plan based on the arrangement of the sorting containers after replenishment. [Appendix 2] the robots include a first robot and a second robot, and a movable range of the first robot and a movable range of the second robot overlap each other; The control device according to claim 1, wherein the motion planning means determines the motion plan including a motion related to the transfer of the item between the first robot and the second robot. [Appendix 3] the sorting container that is not within the movement range of the first robot is within the movement range of the second robot, The control device described in Appendix 2, wherein the motion planning means determines the motion plan including an operation of moving an item that is within the movable range of the first robot and outside the movable range of the second robot by the first robot, into the movable range of the second robot. [Appendix 4] 4. The control device according to any one of claims 1 to 3, further comprising a sorting container control means for controlling the discharge and replenishment of the sorting containers. [Appendix 5] The control device described in Appendix 4, wherein the sorting container control means removes the sorting container to which any of the multiple items is to be sorted and replenishes it with a new sorting container when the sorting container is in a predetermined reference state. [Appendix 6] The control device described in Appendix 4, wherein the sorting container control means, when there is no candidate sorting destination for any of the plurality of items in the plurality of sorting containers, removes any of the plurality of sorting containers and refills a new sorting container. [Appendix 7] The control device according to claim 6, wherein the sorting container control means selects one of the sorting containers from the plurality of sorting containers based on the number or capacity of items contained in each of the plurality of sorting containers. [Appendix 8] The control device according to any one of appendices 1 to 7, wherein the motion planning means generates, as the motion plan, a motion sequence representing the motion of the robot for each time step based on a logical formula that expresses a target task, which is a target to be executed by the robot, using temporal logic. [Appendix 9] 9. The control device of claim 8, wherein the target task is a pick-and-place task of moving the plurality of items to the plurality of sorting containers based on the classification of the items. [Appendix 10] The computer determining a motion plan for the robot with respect to a task of sorting a plurality of articles into a plurality of sorting containers; determining whether the sorting container has been replenished; When the sorting container is replenished, the operation plan is updated based on the arrangement of the sorting container after replenishment. [Appendix 11] determining a motion plan for the robot with respect to a task of sorting a plurality of articles into a plurality of sorting containers; determining whether the sorting container has been replenished; a storage medium storing a program for causing a computer to execute a process of updating the operation plan based on the arrangement of the sorting containers after the sorting containers have been replenished;

[0153] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated herein by reference. [Explanation of symbols]

[0154] 1. 1X control device 2 Warehouse management equipment 3. Communication Network 4. Measurement equipment 5 Controlled equipment 41 Camera 42 BCR 51 Robot 52 Conveying Machinery 53 Sorting container replenishment mechanism 100, 100A sorting control system

Claims

1. a motion planning means for determining a motion plan for a robot for a pick-and-place operation of moving a plurality of articles to a plurality of first sorting containers based on a classification of the articles; and a determination means for determining whether or not a second sorting container associated with a displacement of the first sorting container has been replenished, The operation planning means updates the operation plan based on an arrangement of the first sorting container and the second sorting container after the second sorting container is replenished, the control device.

2. the robots include a first robot and a second robot, and a movable range of the first robot and a movable range of the second robot overlap with each other; The control device according to claim 1 , wherein the motion planning means determines the motion plan including a motion related to a transfer of the article between the first robot and the second robot.

3. the first sorting container that is not within the movable range of the first robot is within the movable range of the second robot, The control device according to claim 2, wherein the motion planning means determines the motion plan including an operation of the first robot moving an item that is within the movable range of the first robot and outside the movable range of the second robot into the movable range of the second robot.

4. 4. The control device according to claim 1, further comprising a sorting container control means for controlling the ejection of the first sorting container and the replenishment of the second sorting container.

5. The control device described in claim 4, wherein the sorting container control means removes the first sorting container and replenishes the second sorting container when the first sorting container to which any one of the plurality of items is to be moved is in a predetermined reference state.

6. The control device described in claim 4, wherein the sorting container control means, when there is no candidate destination for any of the plurality of items in the plurality of first sorting containers, removes any of the plurality of first sorting containers and refills the second sorting container.

7. The control device according to claim 6 , wherein the sorting container control means selects one of the sorting containers from the plurality of first sorting containers based on the number or capacity of articles contained in each of the plurality of first sorting containers.

8. The control device according to any one of claims 1 to 7, wherein the motion planning means generates, as the motion plan, a motion sequence representing the motion of the robot for each time step based on a logical expression that expresses a target task, which is a target to be executed by the robot, using temporal logic.

9. The computer determining a robot motion plan for a pick-and-place operation to move a plurality of articles to a plurality of first sorting bins based on a classification of the articles; determining whether a second sorting container associated with the misalignment of the first sorting container has been replenished; A control method, in which, when the second sorting container is replenished, the operation plan is updated based on an arrangement of the first sorting container and the second sorting container after replenishment.

10. determining a robot motion plan for a pick-and-place operation to move a plurality of articles to a plurality of first sorting bins based on a classification of the articles; determining whether a second sorting container associated with the misalignment of the first sorting container has been replenished; a program that causes a computer to execute a process of updating the operation plan based on an arrangement of the first sorting container and the second sorting container after the second sorting container is replenished, when the second sorting container is replenished;

Images