Waste selection assistance device, waste selection assistance system, and waste selection assistance method

Abstract

To provide a waste selection assistance device, a waste selection assistance system, and a waste selection assistance method capable of suppressing dropping of a waste from a gripper.SOLUTION: A tablet terminal 50 includes a display region 51 (photographed image region 511) in which an image of a beverage container B1 photographed by a camera is displayed, and a first operation designating a first point P1 and a second operation designating a second point P2 on the photographed image region 511 are input. A control device acquires, when the first and second operations are input to the tablet terminal 50, the gripping target position of the beverage container B1 on the basis of a straight line L connecting the first point P1 and the second point P2 in the beverage container B1 in the image, and controls a robot arm so as to grip the gripping target position of the beverage container B1.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to a waste sorting support device, a waste sorting support system, and a waste sorting support method for supporting waste sorting work. [Background technology]

[0002] In recent years, from the viewpoint of effective resource utilization, there is a demand for reuse of waste as recyclable resources. In order to promote reuse of waste, it is necessary to separate the waste by material. For example, demolition waste generated when demolishing houses and facilities, and rubble generated during disasters, contain a mixture of waste of different materials such as wood, metals, glass, and plastics. In addition, beverage containers collected as recyclable waste contain a mixture of PET bottles, aluminum cans, steel cans, and glass bottles.

[0003] The present applicant has disclosed a sorting system for such waste in Patent Document 1. The sorting system disclosed in Patent Document 1 displays on a monitor images of waste transported by a conveyor and information such as the weight of the waste. An operator determines the material of the waste based on the information displayed on the monitor and operates a switch according to the material. The sorting system drives a robot arm, which is a gripping device, to grip the waste on the conveyor and sorts the waste by discharging the waste into a collection box for the material assigned to the operated switch. In addition, the image of the waste is associated with material information indicating its material and stored as training data.

[0004] This sorting system uses stored training data to perform machine learning of waste sorting using artificial intelligence. If the artificial intelligence is able to identify the waste material with sufficient accuracy through machine learning, it will drive a robot arm based on the results of that identification to sort the waste. This makes it possible to automate the waste sorting process. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-109161 A Summary of the Invention [Problem to be solved by the invention]

[0006] In the sorting system of Patent Document 1, when an operator operates a switch in an input section, a robot arm automatically grasps the waste material and discharges it into a collection box for the material that corresponds to that switch.

[0007] Waste comes in a variety of shapes. For example, plastic bottles, which are beverage containers, metal cans (aluminum cans, steel cans), and glass bottles also come in a variety of shapes. Therefore, the robot arm may not be able to properly grasp the waste, and there is a risk that the waste will fall from the robot arm.

[0008] Therefore, an object of the present invention is to provide a waste sorting support device, a waste sorting support system, and a waste sorting support method that can prevent waste from falling from the gripping device. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, a waste sorting support device according to one embodiment of the present invention is characterized in having a display device having a display area in which an image of waste is displayed, an input device for inputting a first operation for designating a first point on the display area and a second operation for designating a second point, and a control device for acquiring a target gripping position for the waste based on a position where a straight line connecting the first point and the second point on the waste in the image passes when the first operation and the second operation are input to the input device.

[0010] In the present invention, it is preferable that the control device displays a first figure at the first point in the display area when the first operation is input to the input device, and displays a second figure at the second point in the display area when the second operation is input to the input device.

[0011] In the present invention, it is preferable that the control device displays a straight line connecting the first point and the second point in the display area when the first operation and the second operation are input.

[0012] In the present invention, it is preferable that an intermediate operation for sequentially specifying points on the display area is interposed between the first operation and the second operation, and that when the intermediate operation is input following the first operation, the control device displays a straight line connecting the first point and the latest point specified in the intermediate operation, and when the second operation is input following the intermediate operation, the control device displays a straight line connecting the first point and the second point.

[0013] In the present invention, it is preferable that the control device controls the gripping device so as to grip the waste at the gripping target position.

[0014] In the present invention, it is preferable that the control device stores in a memory device training data that associates an image of the waste, information regarding a straight line connecting the first point and the second point passing through the waste in the image, and the result of gripping the waste by the gripping device.

[0015] In the present invention, it is preferable that the control device acquires operator information regarding an operator operating the input device and the result of gripping the waste by the gripping device, and associates the gripping result with the operator information.

[0016] In the present invention, it is preferable that the control device acquires operator information regarding the operator operating the input device and the result of gripping the waste by the gripping device, and associates an evaluation result evaluating the gripping result based on predetermined evaluation criteria with the operator information.

[0017] In the present invention, it is preferable that the control device determines a reward for the operator based on the evaluation result.

[0018] In order to achieve the above-mentioned object, a waste sorting support system device according to another embodiment of the present invention is characterized in that it comprises an imaging device for photographing waste, a gripping device for gripping the waste, a display device having a display area in which an image of the waste photographed by the imaging device is displayed, an input device for inputting a first operation for designating a first point on the display area and a second operation for designating a second point, and a control device for obtaining a gripping target position of the waste based on a position through which a straight line connecting the first point and the second point on the waste in the image passes when the first operation and the second operation are input to the input device, and controlling the gripping device to grip the waste at the gripping target position.

[0019] In order to achieve the above-mentioned object, another embodiment of the present invention provides a waste sorting support method, characterized in that, when a first point and a second point on a display area of ​​a display device are specified, a computer obtains a target gripping position for the waste based on a position through which a straight line connecting the first point and the second point on the waste in an image displayed in the display area passes. Effect of the Invention

[0020] According to the present invention, when a worker specifies a first point and a second point while checking an image of the waste material displayed in a display area, a gripping target position of the waste material to be gripped by the gripping device is acquired based on the waste material image and a straight line connecting the first point and the second point. In this way, a more appropriate gripping target position can be specified compared to a configuration in which a predetermined position on the waste material (for example, the center position of the waste material) is set as the gripping target position. Therefore, by having the gripping device grip the gripping target position of the waste material, it is possible to effectively prevent the waste material from falling. [Brief description of the drawings]

[0021] [Figure 1] 1 is a diagram illustrating a schematic configuration of a waste sorting support system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a front view of a tablet terminal of the waste sorting support system. [Diagram 3]FIG. 2 is a functional block diagram of the waste sorting support system. [Figure 4] 4 is a flowchart showing an example of processing in a control device of the waste sorting support system. [Diagram 5] FIG. 1 is a diagram explaining operations input into a tablet terminal (part 1). [Figure 6] FIG. 2 is a diagram explaining operations input into a tablet terminal (part 2). [Figure 7] FIG. 3 is a diagram explaining operations input into a tablet terminal (part 3). [Figure 8] FIG. 4 is a diagram explaining operations input into a tablet terminal (part 4). [Figure 9] FIG. 5 is a diagram explaining operations input into a tablet terminal. [Figure 10] FIG. 6 is a diagram explaining operations input into a tablet terminal. [Figure 11] FIG. 7 is a diagram explaining operations input into a tablet terminal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] A waste sorting support system according to an embodiment of the present invention will be described below with reference to FIGS.

[0023] Fig. 1 is a diagram illustrating a schematic configuration of a waste sorting support system according to an embodiment of the present invention. Fig. 2 is a front view of a tablet terminal of the waste sorting support system. Fig. 3 is a functional block diagram of the waste sorting support system. Fig. 4 is a flowchart showing an example of processing in a control device of the waste sorting support system. Figs. 5 to 11 are diagrams illustrating operations input to the tablet terminal.

[0024] A waste sorting support system 1 according to an embodiment of the present invention supports sorting of beverage containers B, such as, for example, PET bottles, metal cans including aluminum cans and steel cans, and glass bottles. Note that the waste sorting support system 1 can also be used for sorting of other types of waste besides beverage containers.

[0025] As shown in FIG. 1, the waste sorting support system 1 includes a conveyor 10, a robot arm 20, a camera 30, a control device 40, a tablet terminal 50, and a storage device 60.

[0026] The conveyor 10 is a belt-type conveyor. The conveyor 10 has a conveying surface 11. The conveying surface 11 is a place where beverage containers B, which are waste, are placed. The conveyor 10 is a conveying device that conveys the beverage containers B in one direction (conveying direction F). The conveying direction F is parallel to the longitudinal direction of the conveying surface 11. Instead of a belt-type conveyor, other types of conveying devices, such as a roller conveyor or a bucket-type conveyor, may be used as the conveying device.

[0027] An actual position in space on the conveying surface 11 is specified by three-dimensional coordinates (X coordinate, Y coordinate, Z coordinate). The X axis corresponds to the longitudinal direction of the conveying surface 11 of the conveyor 10, the Y axis corresponds to the width direction of the conveying surface 11, and the Z axis corresponds to the height direction of the conveying surface 11.

[0028] The robot arm 20 is a gripping device that grips and lifts a beverage container B on the conveying surface 11. A finger portion 21 for gripping the beverage container B is provided at the tip of the robot arm 20. The robot arm 20 moves the lifted beverage container B to a collection box (not shown) provided for each material. The robot arm 20 is installed beside the conveyor 10. The robot arm 20 can grip a beverage container B in a range 11a (a hatched area in FIG. 1) of the conveying surface 11. As the gripping device, other types of gripping devices, such as a vacuum suction type gripping device equipped with a suction cup, may be used instead of the robot arm 20. In this specification, "grip" means to hold an object regardless of the manner of holding the object, such as grabbing an object or sucking an object.

[0029] The camera 30 is an imaging device that captures images of the beverage container B conveyed by the conveyor 10. The camera 30 outputs image data related to the captured image. The camera 30 includes an RGB camera that captures color images. The camera 30 may include other types of cameras, such as an X-ray camera that captures X-ray images, a 3D camera that captures a subject in three dimensions, or a near-infrared camera that captures images using near-infrared rays. The camera 30 is installed above the conveyor 10. The camera 30 is disposed so as to capture the range 11a of the conveying surface 11 of the conveyor 10 from directly above. The camera 30 may capture video or still images.

[0030] The control device 40 controls the overall operation of the waste sorting support system 1. The control device 40 has a computer. The computer may be a personal computer or a large computer such as a workstation or a server device. The computer has a central processing unit (CPU) and a program storage device consisting of a memory, a hard disk drive, etc., that stores various programs executed by the CPU. The control device 40 controls the conveyor 10, the robot arm 20, and the camera 30.

[0031] The tablet terminal 50 is connected to the control device 40 via a wireless communication line. The tablet terminal 50 has a display area 51, a touch panel 52, and a control unit consisting of a computer (not shown). The tablet terminal 50 is a display device that displays images captured by the camera 30 and an input device into which operations by a worker (operator) are input. The tablet terminal 50 and the control device 40 constitute a waste sorting support device. Note that the waste sorting support system 1 may use a smartphone instead of the tablet terminal 50.

[0032] As shown in FIG. 2, the display area 51 has a captured image area 511 and an operation switch display area 512.

[0033] An image according to image data received from the control device 40 is displayed in the captured image area 511. A position (x coordinate, y coordinate) on the captured image area 511 corresponds to an actual position (X coordinate, Y coordinate) in the space on the transport surface 11.

[0034] Material selection switches 512a-512d and a movement execution switch 512f are displayed in the operation switch display area 512. "PET", "METAL", "GLASS", and "OTHER" are assigned to the material selection switches 512a-512d as materials. The material selection switches 512a-512d and the movement execution switch 512f are displayed in a non-selection color (white) or a selection color (hatched). The non-selection color indicates that the material selection switch is not in an operated state, and the selection color indicates that the material selection switch is in an operated state.

[0035] The touch panel 52 is disposed so as to overlap the display area 51. An operation by an operator's finger is input to the touch panel 52. An operation by an operation pen can also be input to the touch panel 52.

[0036] When the worker touches the finger F1 to a location on the touch panel 52 corresponding to the captured image area 511 (first operation), a point (first point P1) on the captured image area 511 corresponding to the contact location of the finger F1 is specified. When the first point P1 is specified, the tablet terminal 50 transmits information about the first point P1 (first point information Jp1) to the control device 40. The first point information Jp1 includes the x coordinate and the y coordinate of the first point P1.

[0037] Following the first operation, when the worker slides the finger F1 while keeping the finger F1 in contact with the touch panel 52 (middle operation), a point (middle point Px) on the captured image area 511 corresponding to the contact point of the finger F1 is sequentially specified. Each time a middle point Px is specified, the tablet terminal 50 transmits information about the middle point Px (middle point information Jpx) to the control device 40. The middle point information Jpx includes the x coordinate and the y coordinate of the middle point Px.

[0038] Following the intermediate operation, when the worker releases the finger F1 from the touch panel 52 (second operation), a point (second point P2) on the captured image area 511 corresponding to the last contact point of the finger F1 is specified. When the second point P2 is specified, the tablet terminal 50 transmits information about the second point P2 (second point information Jp2) to the control device 40. The second point information Jp2 includes the x coordinate and the y coordinate of the second point P2.

[0039] The first point P1 may be specified by the operator touching the finger F1 to the touch panel 52 and then immediately releasing the finger (tapping) (first operation), and the second point P2 may be specified by tapping the touch panel 52 following the first operation (second operation). In this case, the intermediate point Px is not specified.

[0040] After the first point P1 and the second point P2 are designated, the worker touches the first point P1 on the touch panel 52 or its vicinity (for example, a location within the first figure D1), slides the finger F1 while the finger F1 is in contact with the touch panel 52, and then releases the finger F1 from the touch panel 52 (first point moving operation), and a point on the captured image area 511 corresponding to the last contact location of the finger F1 is designated as a new first point P1. When the new first point P1 is designated, the tablet terminal 50 transmits information (first point moving information Jm1) relating to the new first point P1 to the control device 40. The first point moving information Jm1 includes the x coordinate and the y coordinate of the new first point P1.

[0041] Similarly, after the first point P1 and the second point P2 are designated, the worker touches the second point P2 on the touch panel 52 or a portion near the second point P2 (for example, a portion within the second figure D2) with the finger F1, slides the finger F1 while the finger F1 is in contact with the touch panel 52, and then releases the finger F1 from the touch panel 52 (second point moving operation), and a point on the captured image area 511 corresponding to the last contact point of the finger F1 is designated as a new second point P2. When the new second point P2 is designated, the tablet terminal 50 transmits information (second point moving information Jm2) relating to the new second point P2 to the control device 40. The second point moving information Jm2 includes the x coordinate and the y coordinate of the new second point P2.

[0042] When the worker taps a position on the touch panel 52 corresponding to any of the material selection switches 512a to 512d (material selection operation), the tablet terminal 50 transmits information indicating the material assigned to the material selection switch corresponding to the tapped position (material selection information Jsm) to the control device 40. The tablet terminal 50 displays the material selection switch corresponding to the tapped position in a selected color, and displays the other material selection switches in a non-selected color.

[0043] When the worker taps a position on the touch panel 52 corresponding to the movement execution switch 512f (movement execution operation), information instructing the movement of the beverage container (movement execution information Jex) is transmitted to the control device 40. In response to the tap, the tablet terminal 50 displays the movement execution switch 512f in a selected color. When the tablet terminal 50 receives information indicating the end of the movement (movement end information Jen) from the control device 40, it displays the movement execution switch 512f in a non-selected color.

[0044] Instead of the tablet terminal 50, the waste sorting support system 1 may have a display device (display device) having a liquid crystal display, an organic EL display, or the like, and a pointing device (input device) such as a mouse or a trackball. The display device and pointing device are connected to the control device 40. When a mouse is used as the pointing device, for example, the operation of pressing and holding down the left button of the mouse corresponds to the first operation, the operation of sliding the mouse while holding down the left button corresponds to the intermediate operation, and the operation of releasing the left button corresponds to the second operation.

[0045] The storage device 60 is connected to the control device 40 through a wired communication line. The storage device 60 is, for example, an external hard disk device. The control device 40 and at least one of the tablet terminal 50 and the storage device 60 may be connected via the Internet.

[0046] The computer of the control device 40 executes a program (waste sorting support program) stored in the program storage device, and thereby functions as an image data transmission unit 41, a container detection unit 42, a gripping target position acquisition unit 43, a graphic drawing unit 44, and a movement control unit 45, as shown in FIG. 3.

[0047] The image data transmission unit 41 transmits the image data output from the camera 30 to the tablet terminal 50. The tablet terminal 50 receives the image data from the image data transmission unit 41 and displays it in the captured image area 511. The image data transmitted to the tablet terminal 50 by the image data transmission unit 41 is simply referred to as "transmitted image data."

[0048] The container detection unit 42 detects a beverage container B included in an image related to the transmitted image data, and acquires the position and shape of the beverage container B. When a beverage container B is not included in the image related to the transmitted image data, the container detection unit 42 controls the conveyor 10 to move the conveying surface 11 by the length of the range 11a in the conveying direction F and then stop the conveying surface 11.

[0049] When the gripping target position acquisition unit 43 receives the first point information Jp1 and the second point information Jp2 from the tablet terminal 50, it determines whether or not there is a beverage container B through which the straight line L connecting the first point P1 and the second point P2 passes (i.e., a beverage container B that overlaps with the straight line L) in the image related to the transmitted image data. When there is a beverage container B through which the straight line L passes, the gripping target position acquisition unit 43 determines that the beverage container B is a beverage container B to be moved by the robot arm 20.

[0050] The gripping target position acquisition unit 43 acquires the coordinates (x coordinate, y coordinate on the captured image area 511) of the position G through which the straight line L passes on the beverage container B to be moved in the image, and sets the actual position in the space on the conveying surface 11 corresponding to the position G as the gripping target position Gt (X coordinate, Y coordinate). The gripping target position acquisition unit 43 also estimates the height dimension of the beverage container B to be moved based on the shape of the beverage container B to be moved (acquired by the container detection unit 42), and sets a predetermined height position as the gripping target position Gt (Z coordinate). The predetermined height position is, for example, the center position in the height direction. The gripping target position Gt includes the three-dimensional coordinates of two positions on the surface of the actual beverage container B to be moved.

[0051] When the figure drawing unit 44 receives the first point information Jp1 from the tablet terminal 50, it includes figure data relating to the first figure D1 in the transmission image data so that the first figure D1 is displayed at the first point P1 in the captured image area 511.

[0052] When the figure drawing unit 44 receives the midpoint information Jpx from the tablet terminal 50, the figure drawing unit 44 includes figure data relating to the intermediate figure Dx and the straight line Lx in the transmission image data so that the intermediate figure Dx is displayed at the midpoint Px in the captured image area 511 and the straight line Lx connecting the first point P1 and the midpoint Px is displayed. The figure drawing unit 44 includes only the figure data relating to the intermediate figure Dx and the straight line Lx relating to the latest midpoint Px in the transmission image data, and does not include in the transmission image data the figure data relating to the intermediate figure Dx and the straight line Lx relating to any previous midpoint Px. As a result, only the intermediate figure Dx and the straight line Lx relating to the latest midpoint Px are displayed in the captured image area 511, and the intermediate figure Dx and the straight line Lx relating to any previous midpoint Px are not displayed in the captured image area 511.

[0053] When the figure drawing unit 44 receives the second point information Jp2 from the tablet terminal 50, the figure drawing unit 44 includes figure data related to the second figure D2 and the straight line L in the transmission image data so that the second figure D2 is displayed at the second point P2 in the captured image area 511 and the straight line L connecting the first point P1 and the second point P2 is displayed. When the figure drawing unit 44 receives the second point information Jp2 from the tablet terminal 50, the figure drawing unit 44 does not include figure data related to the intermediate figure Dx and the straight line Lx in the transmission image data. As a result, the intermediate figure Dx and the straight line Lx are not displayed in the captured image area 511. The first figure D1, the second figure D2, and the intermediate figure Dx are circular, but may be other shapes such as a triangle or a rectangle. The first figure D1, the second figure D2, and the intermediate figure Dx are figures of the same shape, but may be figures of different shapes. The straight line Lx and the straight line L are solid lines, but for example, the straight line Lx may be a broken line and the straight line L may be a solid line.

[0054] When the figure drawing unit 44 receives first-point movement information Jm1 from the tablet terminal 50, it updates the first point P1 with the new first point P1. When the figure drawing unit 44 receives second-point movement information Jm2 from the tablet terminal 50, it updates the second point P2 with the new second point P2. The figure drawing unit 44 includes the figure data related to the updated first point P1 and second point P2 in the transmitted image data.

[0055] When the movement control unit 45 receives the material selection information Jsm indicating “PET” from the tablet terminal 50, it selects the PET bottle collection box as the destination of the beverage container B to be moved by the robot arm 20.

[0056] When the movement control unit 45 receives the material selection information Jsm indicating “metal” from the tablet terminal 50, it selects the collection box for metal cans as the destination of the movement of the beverage container B by the robot arm 20.

[0057] When the movement control unit 45 receives the material selection information Jsm indicating “glass” from the tablet terminal 50, it selects the glass bottle collection box as the destination of the beverage container B to be moved by the robot arm 20.

[0058] When the movement control unit 45 receives the material selection information Jsm indicating “other” from the tablet terminal 50, it selects the collection box for other materials as the destination of the movement of the beverage container B by the robot arm 20.

[0059] When the movement control unit 45 receives the movement execution information Jex from the tablet terminal 50, it starts moving the beverage container B to be moved, and grips and lifts the beverage container B to be moved by the robot arm 20, and moves it to the currently selected collection box. When there are multiple beverage containers B to be moved, the movement control unit 45 moves the beverage containers B in the order in which they were selected as the beverage containers to be moved.

[0060] When the movement control unit 45 starts moving the beverage container B to be moved, it transmits information (movement start information Ks) indicating that the beverage container B to be moved has been grasped and lifted from the movement control unit 45 to the figure drawing unit 44. When the figure drawing unit 44 receives the movement start information Ks, it stops including the figure data corresponding to the beverage container B to be moved in the transmitted image data. As a result, the first figure D1, the second figure D2 and the straight line L corresponding to the beverage container B to be moved are no longer displayed (are erased).

[0061] The movement control unit 45 judges whether the beverage container B to be moved has been successfully moved to the currently selected collection box. Specifically, the movement control unit 45 monitors the gripping force (torque) of the robot arm 20 from when the robot arm 20 grips the beverage container B to be moved to the collection box until the robot arm 20 moves the beverage container B to the collection box. When there is no change in the gripping force of the robot arm 20 indicating that the beverage container B has fallen before the beverage container B reaches the collection box, the movement control unit 45 determines the gripping result of the beverage container B to be moved as "success". When there is a change in the gripping force of the robot arm 20 indicating that the beverage container B has fallen before the beverage container B reaches the collection box, the movement control unit 45 determines the gripping result of the beverage container B to be moved as "failure". The movement control unit 45 may obtain the gripping result based on the opening angle of the finger unit 21.

[0062] When the movement control unit 45 finishes moving the beverage container B using the robot arm 20, the container detection unit 42 detects a new beverage container B included in the image related to the transmitted image data, acquires the position and shape of the beverage container B, and prepares to specify the next beverage container B to be moved and acquire the target gripping position Gt for that beverage container B.

[0063] The movement control unit 45 stores in the storage device 60 teacher data that associates image data of the beverage container B to be moved, information on the position G through which the straight line L connecting the first point P1 and the second point P2 in the image related to the image data passes, and the gripping result of the beverage container B to be moved by the robot arm 20. By accumulating this teacher data and performing machine learning, it becomes possible to automatically obtain a gripping target position Gt with a high gripping success probability for the beverage container B based on the image data. Note that the teacher data may be obtained by associating only the image data of the beverage container B to be moved when the gripping result is "successful" with information on the position G through which the straight line L connecting the first point P1 and the second point P2 in the image related to the image data. The teacher data may include material selection information Jsm related to the beverage container B, or may be obtained by associating only the image data of the beverage container B to be moved with the material selection information Jsm related to the beverage container B.

[0064] When a first point P1 and a second point P2 are specified on the captured image area 511 of the tablet terminal 50, the computer of the control device 40 acquires the gripping target position Gt of the beverage container B based on the position G through which a straight line L connecting the first point P1 and the second point P2 on the beverage container B in the image displayed in the captured image area 511 passes.

[0065] The control unit of the tablet terminal 50 may have some of the functions of the control device 40 (for example, the graphics drawing unit 44) or all of the functions.

[0066] Next, an example of processing (waste selection support processing) executed in the control device 40 (computer) of the waste selection support system 1 will be described with reference to the flowchart shown in FIG.

[0067] The control device 40 controls the camera 30 to capture an image of the range 11a of the transport surface 11 (S110). The control device 40 transmits image data relating to the image captured by the camera 30 to the tablet terminal 50 (S120). The control device 40 continues to capture images of the transport surface 11 and transmit the image data while executing the waste sorting support process. Upon receiving the image data from the control device 40, the tablet terminal 50 displays the image relating to the image data in the captured image area 511.

[0068] The control device 40 acquires the position and shape of a beverage container B included in an image related to image data (transmitted image data) transmitted to the tablet terminal 50 (S130). When a beverage container B whose position and shape have been acquired is present, the control device 40 determines that the image related to the transmitted image data includes a beverage container B, and when a beverage container B whose position and shape have been acquired is not present, the control device 40 determines that the image related to the transmitted image data does not include a beverage container B.

[0069] When the image related to the transmitted image data does not include the beverage container B (N in S140), the control device 40 controls the conveyor 10 to move the conveying surface 11 in the conveying direction F by the length of the range 11a, and then stops the conveying surface 11 (S150). Then, the control device 40 again acquires the position and shape of the beverage container B included in the image related to the transmitted image data (S130).

[0070] When the image captured by the camera 30 includes the beverage container B (Y in S140), the control device 40 waits for information to be transmitted from the tablet terminal 50 (S160).

[0071] When the control device 40 receives material selection information Jsm from the tablet terminal 50 in response to a material selection operation input to the tablet terminal 50, the control device 40 selects a collection box corresponding to the material indicated in the material selection information Jsm (S170). The control device 40 waits for the next information to be transmitted from the tablet terminal 50 (S160).

[0072] When the control device 40 receives the first point information Jp1 in response to the input of the first operation to the tablet terminal 50, the control device 40 displays the first figure D1 at the first point P1 on the captured image area 511 based on the first point information Jp1 (S180). The control device 40 waits for the next information to be transmitted from the tablet terminal 50 (S160).

[0073] When the control device 40 receives the midpoint information Jpx in response to an input of an intermediate operation to the tablet terminal 50, it displays an intermediate figure Dx at the midpoint Px on the captured image area 511 based on the midpoint information Jpx, and displays a straight line Lx connecting the first point P1 and the midpoint Px (S190). The control device 40 waits for the next information to be transmitted from the tablet terminal 50 (S160).

[0074] When the control device 40 receives second point information Jp2 in response to input of a second operation to the tablet terminal 50, it displays a second figure D2 at a second point P2 on the captured image area 511 based on the second point information Jp2, and displays a straight line L connecting the first point P1 and the second point P2 (S200).

[0075] When the control device 40 receives the first point information Jp1 and the second point information Jp2, it identifies the beverage container B to be moved based on the first point P1 and the second point P2 (S210). The control device 40 acquires the gripping target position Gt for the beverage container B to be moved based on its shape and the first point P1 and the second point P2 (S220). The control device 40 waits for the next information to be transmitted from the tablet terminal 50 (S160).

[0076] When the control device 40 receives first point movement information Jm1 in response to an input of a first point movement operation to the tablet terminal 50, the control device 40 updates the first point P1 based on the first point movement information Jm1 (S230). The control device 40 displays a first figure D1 at the updated first point P1, and displays a straight line L connecting the updated first point P1 and the second point P2. The control device 40 also identifies a beverage container B to be moved using the updated first point P1, and obtains a gripping target position Gt of the beverage container B to be moved. The control device 40 waits for the next information to be transmitted from the tablet terminal 50 (S160).

[0077] When the control device 40 receives second-point movement information Jm2 in response to input of a second-point movement operation to the tablet terminal 50, it updates the second point P2 based on the second-point movement information Jm2 (S240). The control device 40 displays a second figure D2 at the updated second point P2, and displays a straight line L connecting the first point P1 and the updated second point P2. The control device 40 also identifies the beverage container B to be moved using the updated second point P2, and obtains the grip target position Gt of the beverage container B to be moved. The control device 40 waits for the next information to be transmitted from the tablet terminal 50 (S160).

[0078] When the control device 40 receives movement execution information Jex from the tablet terminal 50 in response to a movement execution operation on the tablet terminal 50, it controls the robot arm 20 to grasp and lift the beverage container B to be moved and move it to the selected collection box (S250). When the control device 40 grasps and lifts the beverage container B to be moved, it erases the first figure D1, the second figure D2 and the straight line L corresponding to the beverage container B to be moved. When the movement of the beverage container B to be moved is completed, the control device 40 acquires the grasping result and transmits movement end information Jen to the tablet terminal 50.

[0079] The control device 40 stores in the storage device 60 teacher data that associates image data of the beverage container B to be moved, information about a position G through which a straight line L connecting the first point P1 and the second point P2 in the image related to the image data passes, and a result of grasping the beverage container B to be moved by the robot arm 20 (S260). The control device 40 repeatedly executes the above processes (S130 to S260).

[0080] Next, an example of an operation input by an operator to the tablet terminal 50 of the waste sorting support system 1 will be described with reference to Figs. 5 to 10. In Figs. 6 to 8, a figure hidden by the finger F1 is indicated by a dashed line. An image of the range 11a of the transport surface 11 is displayed in the captured image area 511 of the tablet terminal 50, and the operator is about to move beverage containers B1 and B2 included in the image.

[0081] The worker estimates that the material of beverage containers B1, B2 is "pet" based on the image displayed in captured image area 511. As shown in FIG. 5, the worker taps a location on touch panel 52 corresponding to material selection switch 512a. As a result, the collection box corresponding to the material "pet" is selected as the collection box to which beverage containers B1, B2 are moved. Note that when control device 40 acquires the position and shape of beverage container B, it may estimate the material of each beverage container B based on the image.

[0082] When the worker touches a portion of the touch panel 52 near the beverage container B1 with the finger F1 (first operation), a first figure D1 is displayed, as shown in Fig. 6. When the worker slides the finger F1 while keeping the finger F1 in contact with the touch panel 52 (intermediate operation), an intermediate figure Dx and a straight line Lx are displayed, as shown in Fig. 7. When the worker removes the finger F1 from the touch panel 52 (second operation), a second figure D2 and a straight line L are displayed, as shown in Fig. 8.

[0083] When the worker inputs a first operation, an intermediate operation, and a second operation for beverage container B1, beverage container B1 through which straight line L passes is selected as the movement target, and a grip target position Gt of beverage container B1 is obtained, as shown in Fig. 9. Similarly, when the worker inputs a first operation, an intermediate operation, and a second operation for beverage container B2, beverage container B2 through which straight line L passes is selected as the movement target, and a grip target position Gt of beverage container B2 is obtained.

[0084] If the worker determines that the position of the first figure D1 (first point P1) is not appropriate, the worker touches the finger F1 to the location of the first figure D1 on the touch panel 52, slides the finger F1 while the finger F1 is in contact with the touch panel 52, and then releases the finger F1 from the touch panel 52 (first point moving operation), thereby moving the first figure D1 and the straight line L to appropriate positions. This updates the gripping target position Gt.

[0085] If the worker determines that the position of the second figure D2 (second point P2) is not appropriate, the worker touches the finger F1 to the location of the second figure D2 on the touch panel 52, slides the finger F1 while the finger F1 is in contact with the touch panel 52, and then releases the finger F1 from the touch panel 52 (first point movement operation), thereby moving the second figure D2 and the straight line L to appropriate positions. This updates the gripping target position Gt.

[0086] 10, the worker taps on a location on the touch panel 52 corresponding to the movement execution switch 512f. This causes the robot arm 20 to grasp and lift up the beverage containers B1 and B2 in order, and move them to the collection box corresponding to the material "PET." In addition, teacher data including the grasping results is stored in the storage device 60. Note that immediately before tapping on the location corresponding to the movement execution switch 512f, the worker may tap on a location on the touch panel 52 corresponding to any of the material selection switches 512a to 512d to select a material.

[0087] Then, as shown in FIG. 11, after the beverage containers B1 and B2 have been moved, the worker performs the same operation on the other beverage containers B to perform sorting.

[0088] As described above, the waste sorting support system 1 includes a robot arm 20 that grasps a beverage container B, a camera 30 that photographs the beverage container B in the range 11a of the transport surface 11, a tablet terminal 50 having a display area 51 (captured image area 511) in which an image of the beverage container B photographed by the camera 30 is displayed, the tablet terminal 50 into which a first operation that designates a first point P1 on the photographed image area 511 and a second operation that designates a second point P2 are input, and a control device 40 that, when the first operation and the second operation are input into the tablet terminal 50, obtains a gripping target position Gt of the beverage container B based on a position G through which a straight line L connecting the first point P1 and the second point P2 on the beverage container B in the image displayed in the photographed image area 511 passes, and controls the robot arm 20 to grasp the beverage container B at the gripping target position Gt.

[0089] As a result, when the worker specifies the first point P1 and the second point P2 while checking the image of the beverage container B displayed in the captured image area 511, the gripping target position Gt of the beverage container B to be gripped by the robot arm 20 is acquired based on the image of the beverage container B and the straight line L connecting the first point P1 and the second point P2. This makes it possible to specify a more appropriate gripping target position Gt compared to a configuration in which a predetermined position on the beverage container B (for example, the center position of the beverage container B) is set as the gripping target position. Therefore, by having the robot arm 20 grip the gripping target position Gt of the beverage container B, it is possible to effectively prevent the beverage container B from falling from the robot arm 20.

[0090] Furthermore, when a first operation is input to the tablet terminal 50, the control device 40 displays a first figure D1 at a first point P1 in the captured image area 511. When a second operation is input to the tablet terminal 50, the control device 40 displays a second figure D2 at a second point P2 in the captured image area 511. In this manner, the first point P1 and second point P2 designated by the worker can be visually grasped, and more appropriate first point P1 and second point P2 can be designated.

[0091] Furthermore, when the first operation and the second operation are input to the tablet terminal 50, the control device 40 displays a straight line L connecting the first point P1 and the second point P2 in the captured image area 511. In this manner, the worker can visually grasp the position through which the straight line L connecting the first point P1 and the second point P2 passes in the image of the beverage container B, and can specify more appropriate first point P1 and second point P2. Note that in this embodiment, the specified first point P1 and second point P2 can be moved by a first point moving operation and a second point moving operation, and more appropriate first point P1 and second point P2 can be specified.

[0092] In addition, an intermediate operation for sequentially specifying points on the captured image area 511 is interposed between the first operation and the second operation. When the intermediate operation is input to the tablet terminal 50 following the first operation, the control device 40 displays a straight line Lx connecting the first point P1 and the latest point (intermediate point Px) specified in the intermediate operation. When the second operation is input to the tablet terminal 50 following the intermediate operation, the control device 40 displays a straight line L connecting the first point P1 and the second point P2. In this manner, the worker can visually grasp the straight line Lx connecting the first point P1 and the intermediate point Px before the second point P2 was specified, and sequentially specify the intermediate point Px while visually checking the position through which the straight line Lx passes in the image of the beverage container B, and then specify the second point P2, thereby allowing more appropriate specification of the first point P1 and the second point P2.

[0093] Furthermore, the control device 40 stores in the storage device 60 training data that associates image data relating to the image of the beverage container B, information about a straight line L connecting a first point P1 and a second point P2 that pass through the beverage container B in the image, and the result of grasping the beverage container B by the robot arm 20. In this way, by performing machine learning using the training data, it becomes possible to automatically obtain a gripping target position Gt with a high probability of successful grasping for the beverage container B based on the image data.

[0094] In the waste sorting support system 1, a worker (operator) who operates the tablet terminal 50 may store operator information that identifies the worker in the storage device 60. The operator information is, for example, a login ID and a password. After logging in to the tablet terminal 50, the worker inputs an operation related to the movement of the beverage container B. The tablet terminal 50 may be located in a remote location, such as outside Japan, and connected to the control device 40 via the Internet.

[0095] In this configuration, the control device 40 may acquire operator information about the worker who operates the tablet terminal 50 and the result of grasping the beverage container B by the robot arm 20, and associate the grasping result with the operator information. In this way, the worker and the grasping result are associated with each other, and the quality of the work of each worker can be confirmed. The operator information and the grasping result are stored in the storage device 60, for example.

[0096] Alternatively, in this configuration, the control device 40 may acquire operator information about the worker who operates the tablet terminal 50 and the gripping result of the beverage container B by the robot arm 20, and may associate an evaluation result obtained by evaluating the gripping result based on a predetermined evaluation criterion with the operator information. The evaluation criterion may be, for example, a success rate of gripping the beverage container B by the robot arm 20, with a success rate of 0 to 20% being graded E, a success rate of 21 to 40% being graded D, a success rate of 41 to 60% being graded C, a success rate of 61 to 80% being graded B, and a success rate of 81 to 100% being graded A. In this way, the quality of the work of the worker can be easily evaluated. The operator information and the evaluation result are stored in the storage device 60, for example.

[0097] The control device 40 may also determine a reward for the worker based on the evaluation result. The reward may be, for example, money or a digital item that is relatively difficult to obtain. This increases the motivation of the worker for the work, leading to an improvement in the quality of the work.

[0098] In the waste sorting support system 1, the control device 40 controls the conveyor 10 to intermittently move the transport surface 11 in the transport direction F. The control device 40 may continuously move the transport surface 11 in the transport direction F. Alternatively, the waste sorting support system 1 may omit the conveyor 10 and support the sorting work of waste placed on the floor of a waste disposal site (floor sorting work).

[0099] The waste sorting support system 1 may also have a plurality of robot arms 20 and a plurality of tablet terminals 50 (or smartphones). In this configuration, the robot arms 20 and the tablet terminals 50 correspond one-to-one. The control device 40 treats a pair of a tablet terminal 50 and a robot arm 20 as one control target unit, and controls the plurality of control target units as independent units. The control device 40 also prevents conflicts from occurring between the control target units. A conflict is, for example, when the same beverage container B is targeted for movement at the same time in multiple control target units, or when the robot arms 20 collide with each other. In this way, multiple workers can simultaneously operate (sort) one waste sorting support system 1.

[0100] Furthermore, in the waste sorting support system 1, the control device 40 continues to photograph the transport surface 11 and transmit image data while the waste sorting support process is being performed. The control device 40 may photograph the transport surface 11 and transmit image data in accordance with the movement of the transport surface 11 and the movement of the beverage container B by the robot arm 20. The control device 40 detects the beverage container B included in the image related to the new transmitted image data, and obtains the position and shape of the beverage container B.

[0101] Although the embodiments of the present invention have been described above, the present invention is not limited to these examples. Any addition, deletion, or design change of components by a person skilled in the art to the above-mentioned embodiments, or any combination of features of the embodiments as appropriate, is also included in the scope of the present invention as long as it does not deviate from the spirit of the present invention. [Explanation of symbols]

[0102] 1. Waste sorting support system 10...Conveyor 11...Transport surface 11a…Range 20…Robot arm 21...Finger part 30…Camera 40...Control device 41...Image data transmission unit 42...Container detection unit 43...Gripping target position acquisition unit 44…Drawing section 45...Movement control unit 50…Tablet device 51...display area 511...Image capture area 512: Operation switch display area 512a~512d…Material selection switch 512f…Move execution switch 52...Touch panel 60…Storage device B…Beverage container, B1…Beverage container, B2…Beverage container P1...1st point, P2...2nd point, Px...intermediate point D1: First figure, D2: Second figure, Dx: Intermediate figure L...straight line, Lx...straight line

Claims

1. a display device having a display area on which an image of waste is displayed; an input device to which a first operation for designating a first point on the display area and a second operation for designating a second point on the display area are input; A waste sorting support device characterized by having a control device that, when the first operation and the second operation are input to the input device, acquires a target gripping position of the waste based on the position where a straight line connecting the first point and the second point on the waste in the image passes.

2. The control device, When the first operation is input to the input device, a first figure is displayed at the first point in the display area; 2. The waste sorting support device according to claim 1, wherein when the second operation is input to the input device, a second figure is displayed at the second point in the display area.

3. The waste sorting support device of claim 1 or claim 2, wherein the control device displays a straight line connecting the first point and the second point in the display area when the first operation and the second operation are input.

4. an intermediate operation for sequentially designating points on the display area is interposed between the first operation and the second operation, The control device, when the intermediate operation is input following the first operation, a straight line connecting the first point and the latest point designated in the intermediate operation is displayed; The waste sorting support device according to claim 3 , wherein when the second operation is input following the intermediate operation, a straight line connecting the first point and the second point is displayed.

5. The waste sorting support device according to claim 1 , wherein the control device controls a gripping device to grip the waste at the gripping target position.

6. The waste sorting support device of claim 5, wherein the control device stores in a memory device training data associating an image of the waste, information regarding a straight line connecting the first point and the second point passing through the waste in the image, and the result of gripping the waste by the gripping device.

7. The control device, Acquire operator information regarding an operator who operates the input device and a result of gripping the waste by the gripping device; The waste sorting support device according to claim 5 , wherein the gripping result is associated with the operator information.

8. The control device, Acquire operator information regarding an operator who operates the input device and a result of gripping the waste by the gripping device; The waste sorting support device according to claim 5 , wherein an evaluation result obtained by evaluating the gripping result based on a predetermined evaluation criterion is associated with the operator information.

9. The waste sorting support device according to claim 8 , wherein the control device determines a reward for the operator based on the evaluation result.

10. A photographing device for photographing waste; A gripping device that grips the waste material; a display device having a display area for displaying an image of the waste photographed by the photographing device; an input device to which a first operation for designating a first point on the display area and a second operation for designating a second point on the display area are input; A waste sorting support system characterized by having a control device that, when the first operation and the second operation are input to the input device, acquires a gripping target position of the waste based on the position where a straight line connecting the first point and the second point on the waste in the image passes, and controls the gripping device to grip the waste at the gripping target position.

11. A waste sorting support method characterized in that, when a first point and a second point on a display area of ​​a display device are specified, a computer obtains a target gripping position for the waste based on the position where a straight line connecting the first point and the second point on the waste in an image displayed in the display area passes.

Images