Processing device, processing system, handling system, processing method, program, or storage medium

Abstract

To provide a processing device, a processing system, a handling system, a processing method, a program, or a storage medium that, when an abnormality occurs in a handling robot, can easily check the state of the handling robot or an article.SOLUTION: A processing device according to an embodiment is configured to: acquire first events each indicating that a grip part of a handling robot has passed one of predetermined route positions; acquire images with the handling robot imaged therein; upon acquiring a second event indicating an abnormality in the handling robot, identify a first period between the first events including an occurrence timing of the second event, or a second period from the first event immediately before the occurrence timing to the occurrence timing; and extract, from the multiple images, the images acquired during the first period or the second period.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a processing device, a processing system, a handling system, a processing method, a program, or a storage medium. [Background technology]

[0002] There is a handling robot that transports objects. When an abnormality occurs in the handling robot, there is a demand for a technology that can easily check the status of the handling robot or the object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6534598 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem that the present invention aims to solve is to provide a processing device, a processing system, a handling system, a processing method, a program, or a storage medium that makes it possible to easily check the status of a handling robot or an article when an abnormality occurs in the handling robot. [Means for solving the problem]

[0005] A processing device according to an embodiment acquires a first event indicating that a gripper of a handling robot has passed a predetermined route position. The processing device acquires images of the handling robot. When the processing device acquires a second event indicating an abnormality in the handling robot, the processing device identifies a first period between the first events including an occurrence timing of the second event, or a second period from the first event immediately before the occurrence timing to the occurrence timing. The processing device extracts the images acquired during the first period or the second period from the multiple images. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram showing a handling system including a processing apparatus according to an embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing an example of a handling robot. [Diagram 3] 3(a) and 3(b) are schematic diagrams for explaining the processing by the handling system according to the embodiment. [Figure 4] 4(a) and 4(b) are schematic diagrams for explaining the processing by the handling system according to the embodiment. [Diagram 5] 5(a) and 5(b) are schematic diagrams for explaining the processing by the handling system according to the embodiment. [Figure 6] FIG. 6 is a schematic diagram for explaining the processing by the processing apparatus according to the embodiment. [Figure 7] FIG. 7 is a schematic diagram for explaining the processing by the processing apparatus according to the embodiment. [Figure 8] FIG. 8 is a schematic diagram for explaining the processing by the processing apparatus according to the embodiment. [Figure 9] FIG. 9 is a schematic diagram showing a display example on a terminal device. [Figure 10] FIG. 10 is a schematic diagram showing a display example on a terminal device. [Figure 11] FIG. 11 is a flowchart illustrating a processing method according to an embodiment. [Figure 12] FIG. 12 is a schematic diagram showing a display example on a terminal device. [Figure 13] FIG. 13 is a schematic diagram for explaining the processing by the processing apparatus according to the embodiment. [Figure 14] FIG. 14 is a schematic diagram for explaining the processing by the processing apparatus according to the embodiment. [Figure 15] FIG. 15 is a schematic diagram showing another example of a handling robot. [Figure 16] FIG. 16 is a schematic diagram showing a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as the actual ones. Even when the same part is shown, the dimensions and ratios of each part may be different depending on the drawing. In this specification and each drawing, elements similar to those already explained are given the same reference numerals, and detailed explanations are omitted as appropriate.

[0008] FIG. 1 is a schematic diagram showing a handling system including a processing apparatus according to an embodiment. As shown in FIG. 1, a handling system 1 according to the embodiment includes a handling robot 10, a control device 20, an imaging device 30, a processing device 40, and a terminal device 50.

[0009] The handling robot 10 includes a gripping unit 15 for gripping an item, and transports the item using the gripping unit 15. The gripping unit 15 grips the item by suction or clamping. In the illustrated example, the handling robot 10 is a vertical articulated picking robot. The picking robot performs a picking operation. More specifically, a picking operation instruction is sent to the picking robot from a host computer. The instruction includes the items to be picked and the number of items. A container containing specified items is placed near the picking robot by a person or another transport robot. The picking robot transports the specified items from the container to another container or device (such as a conveyor belt).

[0010] The control device 20 has functions as an operation control unit 21 and an event collection unit 22. The operation control unit 21 controls the operation of the handling robot 10. Specifically, the operation control unit 21 generates an operation plan and operates the handling robot 10 in accordance with the operation plan. The operation plan includes an article to be transported, a gripping position, a route position, a release position, a gripping force for the article, a gripping method for the article, an article transport speed, and the like.

[0011] The "gripping position" is the position of the gripping unit 15 when an item is gripped. The "intermediate position" is the position through which the gripping unit 15 passes during operation of the handling robot 10. The "release position" is the position at which an item gripped by the gripping unit 15 is released. The "gripping force" is the force required to grip an item being transported. The "gripping method" indicates which method is used to grip an item when the gripping unit 15 has multiple methods for gripping an item. The "transport speed" is the speed of the gripping unit 15 when the handling robot 10 transports an item.

[0012] When generating an operation plan, the operation control unit 21 refers to inventory information registered in the inventory data 25. The inventory information includes the storage location of each item, the inventory quantity of each item, and the like.

[0013] The event collection unit 22 generates and collects events during operation of the handling robot 10. For example, the event collection unit 22 generates an event when the gripper 15 passes through a transit position or when an abnormality occurs in the handling robot 10.

[0014] The photographing device 30 repeatedly photographs the handling robot 10 to obtain images. The photographing device 30 may continuously photograph the handling robot 10 to obtain a video. It is preferable that the photographing device 30 photographs the handling robot 10 from above so that the gripping part 15, the article being transported, and the like can be easily captured.

[0015] The processing device 40 has functions as an acquisition unit 41, a recording unit 42, and an output unit 43. The acquisition unit 41 acquires events generated by the event collection unit 22 and images acquired by the photographing device 30. The recording unit 42 records the acquired images and the acquired events in the storage device 45. The output unit 43 extracts a part of the recorded image according to the acquired event, and outputs it to the terminal device 50.

[0016] The terminal device 50 has functions as a display means 51, an operation means 52, and a correction means 53. The display means 51 is a function for displaying an image output from the output unit 43. The operation means 52 is a function for a user of the terminal device 50 to operate the operation control unit 21. The correction means 53 is a function for a user of the terminal device 50 to correct data of the inventory data 25. For example, the user of the terminal device 50 is an operator who monitors the handling robot 10.

[0017] The control device 20 is connected to the handling robot 10 and the processing device 40 via wired or wireless communication. The terminal device 50 is connected to the control device 20 and the processing device 40 via a network. The photographing device 30 transmits the acquired images to the processing device 40 via wired or wireless communication. Alternatively, the photographing device 30 may store the acquired images in a server on the network, and the processing device 40 may acquire the images stored in the server.

[0018] FIG. 2 is a schematic diagram showing an example of a handling robot. For example, as shown in Fig. 2, the handling robot 10 is a vertical articulated robot. The handling robot 10 includes a manipulator 11 including multiple links 11a and multiple rotation shafts 11b. The links 11a are connected to each other by the rotation shafts 11b. The position and angle of the tip of the manipulator 11 change as each rotation shaft 11b operates. The tip of the manipulator 11 preferably has six degrees of freedom.

[0019] The gripping unit 15 is attached to the tip of the manipulator 11. In the illustrated example, the gripping unit 15 includes a suction mechanism 16 and a clamping mechanism 17.

[0020] The suction mechanism 16 grips an article by suction. Specifically, the suction mechanism 16 includes one or more suction pads 16a. With the suction pad 16a in contact with the article, the pressure inside the suction pad 16a is reduced by a pressure reducing device (not shown). This allows the suction pad 16a to suction the article. The number of suction pads 16a may be more or less than that shown in the illustrated example.

[0021] The clamping mechanism 17 grips an article by pinching. Specifically, the clamping mechanism 17 includes a plurality of rod-shaped support parts 17a. The article is pinched and gripped by the plurality of support parts 17a. The clamping mechanism 17 may include more support parts 17a than the example shown in the figure. The support parts 17a may be configured in a finger shape including one or more joints.

[0022] The gripping unit 15 further includes a switching mechanism 18. The suction mechanism 16 and the clamping mechanism 17 are connected to the switching mechanism 18. The switching mechanism 18 rotates the suction mechanism 16 and the clamping mechanism 17 to switch the mechanism used to grip an article.

[0023] The gripping unit 15 may include only one of the suction mechanism 16 and the clamping mechanism 17, without being limited to the example shown in the drawings. In that case, the switching mechanism 18 is not necessary. Alternatively, the gripping unit 15 may include another gripping mechanism in addition to the suction mechanism 16 and the clamping mechanism 17.

[0024] As an example, a container C1 and a conveyor C2 are placed near the handling robot 10. The handling robot 10 grasps an article A contained in the container C1 and transports it onto the conveyor C2. The conveyor C2 transports the transported article to a predetermined location.

[0025] A detector 19 for detecting the internal state of the container C1 may be provided. The detector 19 detects an article contained in the container C1. The detector 19 includes one or more selected from a camera and a distance measuring sensor. The article is recognized from a color image by the camera or a distance image by the distance measuring sensor. Based on the recognition result, the position, size, shape, etc. of the article are calculated. The position, size, shape, etc. may be calculated by a computer provided in the detector 19 or may be calculated by the control device 20. The detector 19 is provided, for example, above the container C1. Alternatively, the detector 19 may be attached to the manipulator 11.

[0026] When the motion plan is generated, the motion control unit 21 acquires one or more item data selected from the size of the item, the shape of the item, the weight of the item, and the material of the item, and may change at least one selected from the gripping force for the item, the gripping method of the item, and the conveying speed of the item according to the item data. For example, the gripping force is set to be greater as the weight increases. Small items are gripped by clamping, and large items are gripped by suction. Items with smooth surfaces are gripped by suction. Items with uneven surfaces are gripped by clamping. The conveying speed is set to be lower as the item is heavier. By changing the gripping force, the gripping method, or the conveying speed, the item can be gripped more stably.

[0027] 3(a) to 5(b) are schematic diagrams for explaining the processing by the handling system according to the embodiment. When the handling robot 10 transports an article, the operation control unit 21 operates the handling robot 10 to move the gripper 15 to position Po1 above the article A, as shown in Fig. 3(a). Next, as shown in Fig. 3(b), the operation control unit 21 moves the gripper 15 to position Po2 for gripping the article A. The operation control unit 21 causes the gripper 15 to grip the article A at position Po2, and moves the gripper 15 upward.

[0028] The operation control unit 21 moves the gripper 15 to a position Po3 as shown in FIG. 4(a). The position Po3 may be the same as the position Po1. The operation control unit 21 then moves the gripper 15 toward the conveyor C2. At this time, the gripper 15 passes through a pre-calculated position Po4 as shown in FIG. 4(b).

[0029] The operation control unit 21 moves the gripper 15 to position Po5 above the conveyor C2, as shown in Fig. 5(a). Next, the operation control unit 21 moves the gripper 15 to position Po6 for placing the item A, as shown in Fig. 5(b). The operation control unit 21 causes the gripper 15 to release the item A at position Po6, and moves the gripper 15 upward.

[0030] In the series of movements shown in Fig. 3(a) to Fig. 5(b), positions Po1 to Po6 through which the gripping part 15 passes are calculated in advance by the movement control part 21 and are included in the movement plan. The positions Po1 and Po3 to Po5 are examples of pass-through positions. The position Po2 is an example of a gripping position. The position Po6 is an example of a release position.

[0031] In the illustrated example, cameras 31 to 34 are provided above the movement path of the gripper 15. Each of the cameras 31 to 34 is an example of the imaging device 30. The cameras 31 to 34 capture different ranges. Specifically, the camera 31 is provided above the container C1 and captures a range including positions Po1 to Po3. The cameras 32 and 33 capture a range between positions Po3 and Po5. The camera 34 captures a range including positions Po5 and Po6. The cameras 31 to 34 repeatedly acquire images during the operation of the handling robot 10, regardless of whether the gripper 15 is present in the capture range or not.

[0032] FIG. 6 is a schematic diagram for explaining the processing by the processing apparatus according to the embodiment. In the operation shown in Figures 3(a) and 3(b), the gripper 15 reaches a position Po1 at a timing t1, and reaches a position Po2 at a timing t2. Next, in the operation shown in Figures 4(a) and 4(b), the gripper 15 reaches a position Po3 at a timing t3, and reaches a position Po4 at a timing t4. After that, in the operation shown in Figures 5(a) and 5(b), the gripper 15 reaches a position Po5 at a timing t5, and reaches a position Po6 at a timing t6.

[0033] As shown in FIG. 6, the event collection unit 22 collects the passing of the gripper 15 through any of the positions Po1 to Po6 as first events e1 to e6. The first events e1 to e6 are generated at timings t1 to t6, respectively. The camera 31 repeatedly captures images of the gripper 15 while it is in motion, and acquires a plurality of images IMG1 (moving image). Similarly, the cameras 32 to 34 acquire a plurality of images IMG2, a plurality of images IMG3, and a plurality of images IMG4, respectively. The acquisition unit 41 acquires the first events e1 to e6, the plurality of images IMG1, the plurality of images IMG2, the plurality of images IMG3, and the plurality of images IMG4 from the event collection unit 22 and the cameras 31 to 34.

[0034] As shown in FIG. 6, the recording unit 42 may identify periods p0 to p6 between the timings at which first events occurred. The recording unit 42 may assign a label to each acquired image indicating the period during which it was captured. By assigning a label, each image is linked to the period between first events. When the cameras 31 to 34 acquire successive images (video), a common label is assigned to multiple images (parts of the video). The recording unit 42 records the images with the labels in the storage device 45.

[0035] In addition to the first event, the event collection unit 22 generates a second event when an abnormality occurs in the handling robot 10. For example, an "abnormality" refers to the occurrence of an event not included in the operation plan. For example, the second event is generated when an article grasped by the gripper 15 falls or when the gripper 15 comes into contact with an object other than the article to be grasped.

[0036] The content of the abnormality is estimated by the control device 20. For example, as shown in FIG. 2, a negative pressure sensor 16b is provided inside the suction pad 16a. The negative pressure sensor 16b detects the difference (negative pressure) between the pressure in the internal space of the suction pad 16a and the atmospheric pressure. When an article is gripped by the suction mechanism 16, if the pressure detected by the negative pressure sensor 16b falls below a preset threshold, the control device 20 estimates that the article has fallen. When an article is gripped by the clamping mechanism 17, if the force applied to the support portion 17a suddenly decreases, the control device 20 estimates that the article has fallen. The force applied to the support portion 17a is calculated based on the current value of the motor that drives the support portion 17a. Alternatively, a sensor for detecting the force applied to the support portion 17a may be provided. When an impact is detected by the force sensor 15a provided in the handling robot 10, the control device 20 estimates that the gripper portion 15 has come into contact with an object other than the article.

[0037] When a second event indicating a drop of an article is acquired, the recording unit 42 identifies a first period between the first events including the occurrence timing of the second event. When a second event indicating contact of the gripper 15 with an object is acquired, the recording unit 42 identifies a second period from the first event immediately before the occurrence timing of the second event to the occurrence timing of the second event.

[0038] 7 and 8 are schematic diagrams for explaining the processing by the processing apparatus according to the embodiment. As an example, while the gripper 15 moves from position Po3 shown in Fig. 4(a) to position Po4 shown in Fig. 4(b), the article held by the gripper 15 falls. In this case, as shown in Fig. 7, at timing t3' between timing t3 and timing t4, a second event E1 indicating the dropping of the article is generated. Note that even if the article falls, the control device 20 moves the gripper 15 to position Po4.

[0039] The recording unit 42 specifies a first period P1 from a first event e3 (timing t3) immediately before timing t3' to a first event e4 (timing t4) immediately after timing t3'. The recording unit 42 extracts images captured in the first period P1 from the multiple images IMG1 captured by the camera 31. Similarly, the recording unit 42 extracts images captured in the first period P1 from the multiple images IMG2-IMG4 captured by the cameras 32-34.

[0040] As another example, while the gripper 15 moves from position Po3 to position Po4, the gripper 15 comes into contact with another object unintentionally. In this case, as shown in Fig. 8, at timing t3' between timing t3 and timing t4, a second event E2 indicating the contact of the gripper 15 with another object occurs. When the gripper 15 comes into contact with the other object, the control device 20 stops the operation of the handling robot 10.

[0041] The recording unit 42 specifies a second period P2 from a first event e3 (timing t3) immediately before timing t3' to timing t3'. The recording unit 42 extracts images captured during the second period P2 from the multiple images IMG1-IMG4 captured by the cameras 31-34.

[0042] The output unit 43 outputs the one or more extracted images to the terminal device 50. The display means 51 displays the received images so that the user of the terminal device 50 can visually confirm them. The operation means 52 operates the control device 20 in response to an input from the user to the terminal device 50. The correction means 53 corrects the inventory data 25 in response to an input from the user to the terminal device 50. While checking the image displayed by the display means 51, the user can operate the control device 20 or correct the inventory quantity registered in the inventory data 25 using the functions of the operation means 52 or correction means 53.

[0043] 9 and 10 are schematic diagrams showing examples of displays on a terminal device. The terminal device 50 provides the user with functions as an operation means 52 and a correction means 53 through a graphical user interface (GUI). The display means 51 displays, for example, a GUI 100 shown in Fig. 9. The GUI 100 displays images 101-104, live images 111-114, container information 121, item information 122, and icons 131-133. The icons 131 and 132 are examples of the operation means 52.

[0044] Images 101-104 are images captured by cameras 31-34, respectively, at the time when the second event occurs. Here, an example of a case where an object has fallen is shown. Images 101-104 are extracted by recording unit 42. When multiple images are extracted from an image captured by one camera, the multiple images may be displayed while being switched at a predetermined interval. When a camera captures a moving image, images 101-104 may display the moving image extracted by recording unit 42.

[0045] The live images 111-114 are real-time images captured by the cameras 31-34, respectively. For example, one of the live images 111-114 is displayed enlarged compared to the other live images 111-114. The user can select one of the live images 111-114 with the pointer 140 to enlarge the selected image.

[0046] The container information 121 indicates information about the container that the transported goods were stored in. The goods information 122 indicates information about the goods that were transported.

[0047] The icon 131 is displayed to move the arm of the handling robot 10. The user can move the gripper 15 in the direction of the arrow by clicking any of the arrows of the icon 131 with the pointer 140. The icon 132 is displayed to initialize the handling robot 10. When the icon 132 is clicked, the operation of the control device 20 is initialized, and the identification of an abnormality by the control device 20 is canceled. The operation control unit 21 resumes the operation of the handling robot 10 according to the operation plan generated immediately before. The icon 133 is an icon for calling an operator to the installation location of the handling robot 10. For example, when the icon 133 is clicked, a notification is sent to a specific terminal device. In response to the notification, the operator moves to the installation location of the handling robot 10 and examines the handling robot 10.

[0048] 10 in addition to the GUI 100. The GUI 200 displays images 201 to 204, container information 211, item information 212, an icon 221, and an icon 222. The icon 221 and the icon 222 are an example of the correction means 53.

[0049] The images 201-204 are images captured by the cameras 31-34, respectively, at the timing when the second event occurred. The images 201-204 may display a plurality of images or a video that changes sequentially. The container information 211 indicates information about the container that accommodated the transported item. The item information 212 indicates information about the transported item.

[0050] Icons 221 and 222 are displayed to correct the inventory quantity of an item. Icon 221 is selected when an item falls into a container (or conveyor). When an item falls into a source container, the item can be transported to the conveyor by the handling robot 10. When an item falls onto a destination conveyor, the transportation of the item can be considered completed. Therefore, there is no need to correct the inventory quantity of the item. When icon 221 is clicked, the inventory quantity registered in the inventory data 25 is not changed.

[0051] Icon 222 is selected when an item falls outside a container (or conveyor). When an item falls outside a container or conveyor, the item cannot be transported to the conveyor until it is collected and placed in a container. In other words, the inventory of items to be transported is reduced. When icon 222 is clicked, the number of items that fell is subtracted from the inventory recorded in inventory data 25.

[0052] If an item that has fallen into a container or conveyor is damaged, the item cannot be shipped. Therefore, if an item that has fallen into a container or conveyor is damaged, icon 222 is selected and the inventory quantity is reduced. Whether or not an item that has fallen into a container or conveyor is damaged can be confirmed from any of images 201 to 204.

[0053] The user of the terminal device 50 operates the control device 20 or modifies the inventory information of the inventory data 25 as appropriate, while switching between the GUI 100 and the GUI 200.

[0054] FIG. 11 is a flowchart illustrating a processing method according to an embodiment. The processing method M shown in FIG. 11 is executed when a second event occurs in the handling system 1. First, the recording unit 42 determines whether the second event has been acquired by the acquisition unit 41 (step S1). Step S1 is repeated until the second event is acquired. When the second event is acquired, the recording unit 42 determines the content of the abnormality indicated by the second event. For example, the recording unit 42 determines whether the abnormality indicated by the acquired second event is a falling of an article (step S2).

[0055] If the abnormality is a dropped item, the recording unit 42 identifies a first period from a first event immediately before the occurrence timing of the second event to a first event immediately after the occurrence timing (step S11). The recording unit 42 extracts images acquired by the photographing device 30 during the first period (step S12). The output unit 43 outputs the extracted images to the terminal device 50, and the display means 51 displays the images (step S13). The user of the terminal device 50 checks the displayed images (step S14). The user corrects the inventory information of the items using the correction means 53 as necessary (step S15). The user also determines whether the dropped item needs to be collected (step S16). If it is determined that collection is necessary, the user heads to collect the item. Or, the user calls a worker. The user or the worker collects the item and stores it in the warehouse (step S17). If it is determined that collection is not necessary in step S16, or if step S17 is completed, the processing method M ends. The above steps complete the recovery process when an item has fallen.

[0056] If the abnormality is other than the dropping of an article, the recording unit 42 identifies a second period from the first event immediately before the occurrence of the second event to the occurrence of the second event (step S21). The recording unit 42 extracts images captured by the photographing device 30 during the second period (step S22). The output unit 43 outputs the extracted images to the terminal device 50, and the display means 51 displays the images (step S23). The user of the terminal device 50 checks the displayed images (step S24). The user judges from the images whether a worker is needed to recover from the abnormality (step S25). If it is judged that a worker is not needed, the user initializes the handling robot 10 using the operation means 52 (step S26). This causes the operation of the handling robot 10 to resume. If it is judged that a worker is needed, the user calls the worker (step S27). The worker heads to the installation site of the handling robot 10 and checks and recovers the handling robot 10. Completion of step S26 or S27 ends processing method M. Through the above steps, recovery processing in the event of the occurrence of an abnormality other than the dropping of an article is completed.

[0057] Advantages of the embodiment will be described. An abnormality may occur during the operation of the handling robot 10. The abnormality may be an article falling, contact of the gripper 15 with an unintended object, or the like. When an abnormality occurs, it is necessary to check the fallen article and the handling robot 10. Conventionally, when an abnormality occurs, a worker moves to the installation location of the handling robot 10 and checks the article, the handling robot 10, and the like. However, with this method, it takes time for the handling robot 10 to recover from the abnormality.

[0058] Regarding the dropping of an article, it is also possible to resume the operation of the handling robot 10 without checking the details. In that case, however, a discrepancy may occur between the inventory information registered in the inventory data 25 and the actual inventory information. For example, if the inventory quantity in the inventory data 25 is reduced even though the article dropped in the container from which the article was transported or the location of the destination, the inventory quantity in the inventory data 25 will be smaller than the actual inventory quantity. If the inventory quantity in the inventory data 25 is not changed and the article falls in a location other than the container from which the article was transported or the location of the destination, the actual inventory quantity will be smaller than the inventory quantity in the inventory data 25.

[0059] Regarding these problems, the processing device 40 according to the embodiment acquires events from the event collection unit 22 and acquires images from the photographing device 30. The events transmitted from the event collection unit 22 include a first event and a second event. The first event indicates that the gripper 15 of the handling robot 10 has passed a predetermined route position. The second event indicates that an abnormality has occurred in the handling robot 10. When the second event is acquired, the processing device 40 specifies a first period or a second period. The first period is a period between first events including the occurrence timing of the second event. The second period is a period from the first event immediately before the occurrence timing to the occurrence timing. The processing device 40 extracts images acquired in the first period or the second period from the multiple images acquired by the photographing device 30.

[0060] For example, an operator monitoring the handling system 1 can easily check the status of the handling robot 10 or the item from the extracted image. If an item falls, the operator can easily determine by checking the image whether or not the inventory information needs to be corrected. If the gripper 15 comes into contact with an unintended object, the operator can easily determine by checking the image whether or not a worker needs to take action.

[0061] According to the embodiment of the present invention, when an abnormality occurs in the handling robot 10, the state of the handling robot 10 or the object or the like can be easily confirmed.

[0062] FIG. 12 is a schematic diagram showing a display example on a terminal device. When the range is estimated by the recording unit 42 by at least one of the methods described above, the terminal device 50 may display a GUI 100a shown in Fig. 12. The GUI 100a displays an image 102, a live image 112, container information 121, item information 122, and icons 131 to 133. Compared to the GUI 100, the GUI 100a does not include images 101, 103, and 104, and live images 111, 113, and 114.

[0063] 12, it is estimated that the gripper 15 was within the shooting range of the camera 32 when the article fell. As a result, the image 102 captured by the camera 32 is extracted and displayed from the images 101-104 captured by the cameras 31-34, respectively. In addition, a real-time live image 112 captured by the camera 32 is displayed.

[0064] The processing device 40 may display container information and article information on the terminal device 50 in addition to the extracted image, as shown in Figures 9, 10, and 12. This allows the operator to easily understand which article during transportation an abnormality occurred.

[0065] The terminal device 50 preferably has a function of an operation means 52. By providing the terminal device 50 with the function of the operation means 52, the operator can initialize the operation of the handling robot 10 in accordance with the result of the abnormality confirmation without moving to the installation location of the handling robot 10. Furthermore, the terminal device 50 preferably has a function of a correction means 53. By providing the terminal device 50 with the function of the correction means 53, the operator can appropriately correct the inventory information registered in the inventory data 25 in accordance with the result of the abnormality confirmation. These functions can further improve the convenience of the handling system 1.

[0066] The processing system 2 (shown in FIG. 1) including the imaging device 30, the processing device 40, and the terminal device 50 may be retrofitted to an existing handling robot 10 and control device 20. If the imaging device 30 for photographing the handling robot 10 is already installed, the retrofitting of the imaging device 30 may be omitted. Retrofitting the processing system 2 to an existing handling robot 10 and control device 20 facilitates recovery processing in the event of an abnormality occurring in the handling robot 10.

[0067] When images are acquired by a plurality of image capturing devices 30, the recording unit 42 may estimate the range in which the gripping unit 15 was present at the occurrence timing of the second event from the image capturing range of each image capturing device 30. For example, coordinate data indicating the image capturing range of each image capturing device 30 is registered in advance in the storage device 45. The recording unit 42 estimates the range in which the gripping unit 15 was present when the second event occurred based on the transit position of the gripping unit 15 included in the operation plan, the received first event, and the like. In this case, the recording unit 42 extracts images from the image capturing device 30 that captures the estimated range among the plurality of image capturing devices 30. According to this method, at the time of occurrence of the second event, images other than the gripping unit 15 are difficult to extract, and images of the gripping unit 15 are easy to extract. The number of images that the operator needs to check is reduced, improving convenience.

[0068] When the recording unit 42 acquires a second event indicating the falling of an object, the recording unit 42 may estimate the range in which the fallen object exists from the imaging range of each imaging device 30. For example, as shown in FIG. 2, the gripping unit 15 includes an acceleration sensor 15b. The acceleration sensor 15b detects acceleration generated in the gripping unit 15. When an object falls, a force is generated in the gripping unit 15 in a direction opposite to the falling direction of the object. The acceleration sensor 15b detects acceleration caused by this force. The direction in which the object fell can be estimated from the data detected by the acceleration sensor 15b. The recording unit 42 estimates the range in which the fallen object exists from the imaging range of each imaging device 30. The recording unit 42 extracts an image from the imaging device 30 that captures the estimated range among the multiple imaging devices 30. According to this method, it becomes easier to extract an image in which the fallen object is captured. The number of images that the operator needs to check is reduced, and the convenience of the handling system 1 can be improved.

[0069] When the range of the dropped object is estimated, a sensor other than an acceleration sensor may be used. For example, a pressure sensor is provided below the operating range of the gripping unit 15. A plurality of mat-type pressure sensors are arranged, and each pressure sensor detects the force applied to the mat. When an object falls and pressure is detected by any of the pressure sensors, it is estimated that the object has fallen on the mat of that pressure sensor. The recording unit 42 extracts an image from the imaging device 30 that captures the estimated mat, among the plurality of imaging devices 30.

[0070] 13 and 14 are schematic diagrams for explaining the processing by the processing apparatus according to the embodiment. 13, cameras 31 to 33 are installed at the installation location of the handling robot 10. The cameras 31 to 33 capture images of at least one of the handling robot 10, the container C1, and the conveyor C2 from different directions. The cameras 31 to 33 are preferably installed so as to cover each other's blind spots.

[0071] A plurality of virtual sections are set in advance in the installation location of the handling robot 10. For example, as shown in FIG. 14, 24 sections S (4×6) are set in the installation location. 11 ~S 46 are set. The multiple sections are aligned along the X and Y directions. Each section is associated with one of the cameras 31 to 33.

[0072] When the recording unit 42 acquires a second event indicating a drop of the article, the recording unit 42 acquires an acceleration α X and acceleration α in the Y direction Y The recording unit 42 acquires the acceleration α X and acceleration α Y By integrating each of these over time, we obtain the velocity of the falling object in the X direction, v X and the velocity in the Y direction v Y Calculate.

[0073] Next, the recording unit 42 calculates the falling time t of the article. For example, a height is set in advance for each section. For a section where the container C1, the conveyor C2, etc. are not placed, the height of the floor surface is set. For a section where the container C1 is placed, the height of the bottom surface of the container C1 is set. For a section where the conveyor C2 is placed, the height of the top surface of the conveyor C2 is set.

[0074] The recording unit 42 refers to the height h1 of the gripping unit 15 when the article fell. The recording unit 42 also identifies the section in which the gripping unit 15 was located when the article fell, and refers to the height h2 of that section. The recording unit 42 calculates the falling time t of the article based on the height h1 of the gripping unit 15, the height h2 of the section, and the gravitational acceleration g.

[0075] The recording unit 42 records the position p1 of the gripping unit 15 in the X direction when the article is dropped. X and the position p1 of the gripping portion 15 in the Y direction when the article falls. Y The recording unit 42 refers to the velocity v in the X direction. X By integrating this with the falling time t of the object, the distance traveled in the X direction during the fall dX The recording unit 42 calculates the velocity v Y By integrating the distance traveled by the object in the Y direction during the fall, d Y Calculate.

[0076] The recording unit 42 is positioned at a position p1 in the X direction. X , position p1 in the Y direction Y , the distance traveled in the X direction d X , and the travel distance in the Y direction d Y From the position p2 of the dropped object in the X direction X and position p2 in the Y direction Y The recording unit 42 calculates the position of the item (p2 X , p2 Y The recording unit 42 determines which section includes the image captured by the selected camera. The recording unit 42 selects a camera associated with the determined section. The recording unit 42 extracts an image captured by the selected camera.

[0077] In the example shown in FIGS. 13 and 14, the item is in the section S 33 It is determined that the object fell into the section S. 33 is linked to the camera 32. The recording unit 42 extracts the images captured by the camera 32.

[0078] FIG. 15 is a schematic diagram showing another example of a handling robot. The handling robot to which the embodiment of the present invention is applied may be the picking robot described above, or may be a robot other than a picking robot. For example, the embodiment of the present invention may be applied to a handling robot 60 shown in FIG.

[0079] The handling robot 60 is an automated crane. The handling robot 60 includes a support 61, a rail 62, a hoist 63, a clam 64 (gripping portion), and a detector 65. The support 61 is a rod-shaped member erected in the vertical direction. The rail 62 is provided along the horizontal direction and is hung on the multiple support 61. The rail 62 can slide the hoist 63 in the horizontal direction. The clam 64 is hung on the hoist 63. The hoist 63 can move the clam 64 in the vertical direction.

[0080] The handling robot 60 grasps the article contained in the container C3 with the clam 64 and transports it to the container C4. Instead of the clam 64, a hook or the like may be provided on the handling robot 60. The detector 65 detects the article contained in the container C3. The detector 65 includes one or more selected from a camera and a distance measuring sensor.

[0081] The handling robot 60 is automatically operated by the operation control unit 21 shown in FIG. 1. The event collection unit 22 generates a first event every time a clam 64 passes a predetermined route position. In addition, the event collection unit 22 generates a second event when an abnormality occurs in the handling robot 60. The photographing device 30 photographs the handling robot 60. The processing device 40 extracts an image in response to acquisition of the second event. When an abnormality occurs in the handling robot 60, the operator can check the extracted image on the terminal device 50.

[0082] The embodiments of the present invention are particularly suitable for picking robots. The size, shape, weight, and the like of items transported by picking robots vary greatly from item to item. In addition, items transported by picking robots are generally stored irregularly in containers. That is, the position of an item in the container from which it is transported is different each time. For this reason, abnormalities occur more frequently in picking robots than in other handling robots. By applying the embodiments to picking robots, recovery processing can be made easier and the operating efficiency of the picking robot can be improved.

[0083] One terminal device 50 may be connected to multiple processing devices 40 via a network. The multiple processing devices 40 acquire events from the multiple control devices 20, respectively. This allows multiple handling robots 10 to be monitored by one terminal device 50.

[0084] FIG. 16 is a schematic diagram showing a hardware configuration. 16 is used for each of the control device 20, the processing device 40, and the terminal device 50. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0085] The ROM 92 stores a program for controlling the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to realize each of the above-mentioned processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0086] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a working memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During execution of the program, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0087] The storage device 94 stores data necessary for executing the programs and data obtained by executing the programs.

[0088] The input interface (I / F) 95 can connect the computer 90 and the input device 95a. The input I / F 95 is, for example, a serial bus interface such as USB. The CPU 91 can read various data from the input device 95a via the input I / F 95.

[0089] The output interface (I / F) 96 can connect the computer 90 and the output device 96a. The output I / F 96 is, for example, a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HPMI (registered trademark)). The CPU 91 can transmit data to the output device 96a via the output I / F 96 and cause the output device 96a to display a user interface.

[0090] The communication interface (I / F) 97 can connect the computer 90 to a server 97a outside the computer 90. The communication I / F 97 is, for example, a network card such as a LAN card. The CPU 91 can read various data from the server 97a via the communication I / F 97.

[0091] The storage device 94 includes one or more selected from a hard disk drive (HDD) and a solid state drive (SSD). The input device 95a includes one or more selected from a mouse, a keyboard, a microphone (audio input), and a touch pad. The output device 96a includes one or more selected from a monitor, a projector, a printer, and a speaker. A device having the functions of both the input device 95a and the output device 96a, such as a touch panel, may be used.

[0092] Each process executed by the control device 20, the processing device 40, or the terminal device 50 may be realized by one computer 90 or by a plurality of computers 90 working together.

[0093] The various data processing operations described above may be recorded as a program that can be executed by a computer on a magnetic disk (such as a flexible disk or a hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0094] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, the computer reads a program from the recording medium and causes a CPU to execute instructions described in the program based on the program. The computer may acquire (or read) the program via a network.

[0095] Embodiments of the invention include the following features. (Feature 1) Acquire a first event indicating that a gripping unit of the handling robot has passed a predetermined intermediate position; Acquire an image of the handling robot; When a second event indicating an abnormality of the handling robot is acquired, a first period between the first events including an occurrence timing of the second event, or a second period from the first event immediately before the occurrence timing to the occurrence timing, is identified; A processing device that extracts the image acquired during the first period or the second period from the plurality of images. (Feature 2) A plurality of ranges are photographed by a plurality of photographing devices, When the second event is acquired, the range in which the gripping part was present at the occurrence timing is estimated among the multiple ranges; and The processing device according to feature 1, further comprising: extracting the image from the imaging device that captures the estimated range, among the plurality of imaging devices. (Feature 3) A plurality of ranges are photographed by a plurality of photographing devices, When the second event indicating a drop of the article gripped by the gripping unit is acquired, the range in which the dropped article is present is estimated among the multiple ranges; The processing device according to feature 1, further comprising: extracting the image from the imaging device that captures the estimated range, among the plurality of imaging devices. (Feature 4) A plurality of ranges are photographed by a plurality of photographing devices, A plurality of sections are set in the installation location of the handling robot, Each of the plurality of sections is associated with one of the plurality of photographing devices; When the second event indicating a drop of the article gripped by the gripping unit is acquired, the section into which the article has fallen is determined; The processing device according to feature 1, extracting the image from the imaging device associated with the determined section from among the plurality of imaging devices. (Feature 5) the second event is generated when an article gripped by the gripper falls or when the gripper comes into contact with an object other than the article to be gripped; If the article has fallen, extracting the image for the first period; 5. The processing device according to any one of features 1 to 4, wherein, when the gripping portion comes into contact with the object, the image for the second period is extracted. (Feature 6) A processing device according to any one of features 1 to 5; a terminal device connected to the processing device via a network, The processing device transmits the extracted image to the terminal device. (Feature 7) The processing system according to feature 6, further comprising an imaging device for photographing the handling robot to obtain the image. (Feature 8) A processing device according to any one of features 1 to 5; an imaging device for capturing an image of the handling robot; The handling robot; A control device that controls the handling robot so that the gripping unit passes through the intermediate position. (Feature 9) A terminal device connected to the processing device via a network, The terminal device displaying a user interface including the extracted image, an item to be transported by the handling robot, and a means for modifying inventory information of the item; 9. A handling system as described in feature 8, further comprising: a means for modifying the inventory information of the item in response to an input from a user to the modification means. (Feature 10) A terminal device connected to the processing device via a network, The terminal device displaying a user interface including the extracted image and an operation means for initializing an operation of the handling robot; 9. The handling system according to feature 8, further comprising: initializing an operation of the handling robot in response to an input from a user to the operating means. (Feature 11) The control device includes: Acquire one or more pieces of article data selected from a size of an article to be grasped, a shape of the article, a weight of the article, and a material of the article; 11. The handling system according to any one of features 8 to 10, wherein at least one selected from a gripping force applied to the object, a gripping method for the object, and a conveying speed for the object are changed according to the object data. (Feature 12) 12. The handling system according to any one of Features 8 to 11, wherein the handling robot is a picking robot. (Feature 13) The computer Acquire a first event indicating that a gripping unit of the handling robot has passed a predetermined intermediate position; Acquire an image of the handling robot; When a second event indicating an abnormality of the handling robot is acquired, a first period between the first events including an occurrence timing of the second event, or a second period from the first event immediately before the occurrence timing to the occurrence timing is identified; extracting the image acquired during the first period or the second period from the plurality of images; Processing methods. (Feature 14) the second event is generated when an article gripped by the gripper falls or when the gripper comes into contact with an object other than the article to be gripped; When the article is dropped, the computer extracts the images for the first time period; 14. The processing method of claim 13, wherein the computer extracts the images for the second period when the gripper comes into contact with the object. (Feature 15) A plurality of ranges are photographed by a plurality of photographing devices, A plurality of sections are set in the installation location of the handling robot, Each of the plurality of sections is associated with one of the plurality of photographing devices; The computer includes: When the second event indicating a drop of the article gripped by the gripping unit is acquired, the section into which the article has fallen is determined; extracting the image from the imaging device associated with the determined section from among the plurality of imaging devices; 15. The method according to claim 13 or 14. (Feature 16) 16. A program for causing the computer to execute the processing method according to any one of features 13 to 15. (Feature 17) A storage medium storing the program according to feature 16.

[0096] According to the embodiments described above, a processing device, a processing system, a handling system, a processing method, a program, or a storage medium is provided that can easily check the status of the handling robot 10 or an item, etc., when an abnormality occurs in the handling robot 10.

[0097] As used herein, "or" indicates that "at least one or more" of the items listed in the sentence may be employed.

[0098] Although some embodiments of the present invention have been illustrated above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. In addition, the above-mentioned embodiments can be implemented in combination with each other. [Explanation of symbols]

[0099] 1: handling system, 2: processing system, 10: handling robot, 11: manipulator, 11a: link, 11b: rotation axis, 15: gripping section, 15a: force sensor, 15b: acceleration sensor, 16: suction mechanism, 16a: suction pad, 16b: negative pressure sensor, 17: clamping mechanism, 17a: support section, 18: switching mechanism, 19: detector, 20: control device, 21: operation control section, 22: event collection section, 25: inventory data, 30: photographing device, 31-34: camera, 40: processing device, 41: acquisition section, 42: recording section, 43: output section, 45: storage device, 50: terminal device, 51: display means, 52: operation means, 53: correction means, 60: handling robot, 61: support, 62: rail, 63: hoist, 64: clam, 65: detector, 90: computer, 91: CPU, 92: ROM, 93: RAM, 94: storage device, 95: input interface, 95a: input device, 96: output interface, 96a: output device, 97: communication interface, 97a: server, 98: system bus, 101-104: image, 111-114: live image, 121: container information, 122: item information, 131-133: icon, 140: pointer, 201-204: image, 211: container information, 212: item information, 221, 222: icon, A: item, C1: container, C2: conveyor, C3, C4: container, e1-e6: first event, E1, E2: second event, IMG1 to IMG4: images, M: processing method, p0 to p6: period, P1: first period, P2: second period, t1 to t6, t3': timing

Claims

1. Acquire a first event indicating that a gripping unit of the handling robot has passed a predetermined intermediate position; Acquire an image of the handling robot; When a second event indicating an abnormality of the handling robot is acquired, a first period between the first events including an occurrence timing of the second event, or a second period from the first event immediately before the occurrence timing to the occurrence timing is identified; A processing device that extracts, from the plurality of images, the images acquired during the first period or the second period.

2. A plurality of ranges are photographed by a plurality of photographing devices, When the second event is acquired, the range in which the gripping portion was present at the occurrence timing is estimated from among the plurality of ranges; The processing device according to claim 1 , further comprising: a processor configured to extract the image from the imaging device that captures the estimated range, out of the plurality of imaging devices.

3. A plurality of ranges are photographed by a plurality of photographing devices, When the second event indicating a drop of the article gripped by the gripping unit is acquired, the range in which the dropped article is present is estimated among the multiple ranges; The processing device according to claim 1 , further comprising: a processor configured to extract the image from the imaging device that captures the estimated range, out of the plurality of imaging devices.

4. A plurality of ranges are photographed by a plurality of photographing devices, A plurality of sections are set in the installation location of the handling robot, Each of the plurality of sections is associated with one of the plurality of photographing devices; When the second event indicating a drop of the article gripped by the gripping unit is acquired, the section into which the article has fallen is determined; The processing method according to claim 1 , further comprising extracting the image from the imaging device associated with the determined section from among the plurality of imaging devices.

5. the second event is generated when an article gripped by the gripper falls or when the gripper comes into contact with an object other than the article to be gripped; If the object falls, extracting the image for the first period; The processing device according to claim 1 , further comprising: a processor configured to extract the image for the second period when the gripping portion comes into contact with the object.

6. A processing device according to claim 1 ; a terminal device connected to the processing device via a network, The processing device transmits the extracted image to the terminal device.

7. The processing system according to claim 6 , further comprising an image capturing device that captures an image of the handling robot to obtain the image.

8. A processing device according to claim 1 ; an imaging device for capturing an image of the handling robot; The handling robot; A control device that controls the handling robot so that the gripping unit passes through the intermediate position.

9. A terminal device connected to the processing device via a network, The terminal device displaying a user interface including the extracted image, an item to be transported by the handling robot, and a means for modifying inventory information of the item; The handling system according to claim 8 , further comprising: a correction unit that changes the inventory information of the article in response to an input from a user to the correction unit.

10. A terminal device connected to the processing device via a network, The terminal device displaying a user interface including the extracted image and an operation means for initializing an operation of the handling robot; 9. The handling system according to claim 8, wherein an operation of the handling robot is initialized in response to an input from a user to the operation means.

11. The control device includes: Acquire one or more item data selected from a size of an item to be grasped, a shape of the item, a weight of the item, and a material of the item; The handling system according to any one of claims 8 to 10, wherein at least one selected from a gripping force applied to the item, a gripping method for the item, and a conveying speed for the item is changed depending on the item data.

12. The handling system according to any one of claims 8 to 10, wherein the handling robot is a picking robot.

13. The computer Acquire a first event indicating that a gripping unit of the handling robot has passed a predetermined intermediate position; Acquire an image of the handling robot; When a second event indicating an abnormality of the handling robot is acquired, a first period between the first events including an occurrence timing of the second event, or a second period from the first event immediately before the occurrence timing to the occurrence timing is identified; extracting the image acquired during the first period or the second period from the plurality of images; Processing methods.

14. the second event is generated when an article gripped by the gripper falls or when the gripper comes into contact with an object other than the article to be gripped; When the object is dropped, the computer extracts the images for the first period of time; The processing method according to claim 13 , wherein the computer extracts the images for the second period when the gripper comes into contact with the object.

15. A plurality of ranges are photographed by a plurality of photographing devices, A plurality of sections are set in the installation location of the handling robot, Each of the plurality of sections is associated with one of the plurality of photographing devices; The computer includes: When the second event indicating a drop of the article gripped by the gripping unit is acquired, the section into which the article has fallen is determined; extracting the image from the imaging device associated with the determined section from among the plurality of imaging devices; The method of claim 13.

16. A program for causing a computer to execute the processing method according to any one of claims 13 to 15.

17. A storage medium storing the program according to claim 16.

Images