Material handling equipment

Abstract

This object-processing device (1) comprises: a first transport unit (2) for transporting a plurality of objects along a first transport path (5); a first imaging unit (31) for capturing a first image in which the first transport path (5) is reflected; a robot (32) for causing a first object selected from among the plurality of objects to move from the first transport path (5) to a second transport path (21) on the basis of the first image using a trained model; a second transport unit (3) for transporting the first object along the second transport path (21); a second imaging unit (35) for capturing a second image in which a second object from among the plurality of objects is reflected, the second object having moved from the first transport path (5) or the second transport path (21) to an imaging region (39); a label information generation unit (34) for generating label information assigned to the second object; and a learning unit (48) that updates the trained model on the basis of the second image and the label information.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to an object processing device. [Background technology]

[0002] An automatic recyclable waste sorting device is known that automatically separates recyclable waste from multiple pieces of waste transported along a transport path. The automatic recyclable waste sorting device uses a learning model created by machine learning to determine whether multiple pieces of waste are recyclable waste based on images of the multiple pieces of waste. The learning model is updated by additionally training the learning model using newly created training data, and the automatic recyclable waste sorting device can improve the accuracy of determining whether multiple pieces of waste are recyclable waste by using the updated learning model (Japanese Patent Publication No. 2019-533570, Japanese Patent No. 6854995, and International Publication No. 2016 / 084336). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2019-533570 [Patent Document 2] Patent No. 6854995 [Patent Document 3] International Publication No. 2016 / 084336 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, such training data is created using missorted waste that has been missorted by an automatic recyclable waste sorting device, which increases the burden on workers who handle the missorted waste.

[0005] The disclosed technology has been developed in consideration of these points, and aims to provide an object processing device that facilitates the work of workers when creating training data to be used for additional learning of a learning model created by machine learning. [Means for solving the problem]

[0006] An object processing apparatus according to one embodiment of the present disclosure includes a first conveying unit that conveys a plurality of objects along a first conveying path, a first imaging unit that captures a first image that shows the first conveying path, a robot that moves a first object selected from the plurality of objects based on the first image from the first conveying path to a second conveying path using a learning model, a second conveying unit that conveys the first object along the second conveying path, a second imaging unit that captures a second image that shows a second object of the plurality of objects that has been moved from the first conveying path or the second conveying path to an imaging area, a label information generation unit that generates label information to be assigned to the second object, and a learning unit that updates the learning model based on the second image and the label information. [Effects of the Invention]

[0007] The disclosed object processing device can simplify the work of an operator when creating training data to be used for additional learning of a learning model created by machine learning. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a plan view showing an automatic recyclable waste sorting apparatus provided with an object processing apparatus according to a first embodiment. [Figure 2] FIG. 2 is a side view showing the reversing device. [Figure 3] FIG. 3 is a block diagram showing an object processing device. [Figure 4] FIG. 4 is a plan view showing a worker monitoring multiple pieces of trash placed in a first worker trash removal area and a second worker trash removal area. [Figure 5]FIG. 5 is a plan view showing the operation of a worker placing non-target waste B in the worker waste supply area. [Figure 6] FIG. 6 is a plan view showing the operation of a worker removing recyclable waste A to be processed from the first transport path in the first worker waste removal area. [Figure 7] FIG. 7 is a flowchart showing a part of the operation of updating the learning model. [Figure 8] FIG. 8 is a side view showing the recyclable waste to be processed gripped by the inverting device. [Figure 9] FIG. 9 is a side view showing the recyclable waste to be treated that has been turned over by the turning device. [Figure 10] FIG. 10 is a side view showing the recyclable waste to be processed that has been placed in the incorrectly separated waste image capturing area by the inverting device. [Figure 11] FIG. 11 is a plan view showing an object processing apparatus according to a second embodiment. [Figure 12] FIG. 12 is a plan view showing an object processing apparatus according to a third embodiment. [Figure 13] FIG. 13 is a plan view showing an object processing apparatus according to a fourth embodiment. [Figure 14] FIG. 14 is a side view showing an image capturing area of ​​the missorted waste in the object processing apparatus of the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] An object processing apparatus according to an embodiment of the present disclosure will be described below with reference to the drawings. Note that the following description does not limit the technology of the present disclosure. In the following description, the same components are given the same reference numerals, and duplicated descriptions will be omitted. [Example]

[0010] As shown in FIG. 1, the object processing device 1 of the first embodiment is provided in an automatic recyclable waste sorting device 10. FIG. 1 is a plan view showing the automatic recyclable waste sorting device 10 in which the object processing device 1 of the first embodiment is provided. The automatic recyclable waste sorting device 10 includes a first conveying device 2, a second conveying device 3, and the object processing device 1. The first conveying device 2 is formed of a belt conveyor. A first conveying path 5 is formed in the first conveying device 2. The first conveying path 5 extends along a horizontal plane, and the straight line along which the first conveying path 5 extends is parallel to the conveying direction 6. The first conveying path 5 includes a waste supply area 11, a sorting image capturing area 12, a robot waste removal area 14, a first worker waste removal area 15, and a first incorrectly sorted waste supply area 16.

[0011] The sorting image capturing area 12 is located downstream in the conveying direction 6 of the waste supply area 11, and the robot waste removal area 14 is located downstream in the conveying direction 6 of the sorting image capturing area 12. The first worker waste removal area 15 is located downstream in the conveying direction 6 of the robot waste removal area 14. The first incorrectly sorted waste supply area 16 is located downstream in the conveying direction 6 of the first worker waste removal area 15. The first conveying device 2 conveys objects to be placed on the first conveying path 5 along the first conveying path 5 in the conveying direction 6, and ultimately places the objects placed on the first conveying path 5 in a non-target waste disposal area.

[0012] The second conveying device 3 is formed of a belt conveyor. A second conveying path 21 is formed in the second conveying device 3. The second conveying path 21 is along a horizontal plane, and the straight line along which the second conveying path 21 runs is parallel to the conveying direction 6. The second conveying path 21 includes a robot waste supply area 22, a second worker waste removal area 23, and a second incorrectly sorted waste supply area 24. The second worker waste removal area 23 is located downstream of the robot waste supply area 22 in the conveying direction 6. The second incorrectly sorted waste supply area 24 is located downstream of the second worker waste removal area 23 in the conveying direction 6. The second conveying device 3 conveys objects placed on the second conveying path 21 along the second conveying path 21 in the conveying direction 6, and ultimately places objects placed on the first conveying path 5 in a recyclable waste bin.

[0013] The object processing apparatus 1 includes a sorting imaging unit 31, a robot 32, a third conveying device 33, a label information detection camera 34, an annotation imaging unit 35, and a control device 36. The sorting imaging unit 31 is disposed near the sorting image capturing area 12. The sorting imaging unit 31 is controlled by the control device 36 to capture an image of an object placed in the sorting image capturing area 12. The robot 32 is disposed near the robot waste removal area 14 and near the robot waste supply area 22. The robot 32 is controlled by the control device 36 to remove an object placed in the robot waste removal area 14 from the first conveying path 5 and place it in the robot waste supply area 22 of the second conveying path 21.

[0014] The third conveying device 33 is formed of a belt conveyor. A third conveying path 37 is formed in the third conveying device 33. The third conveying path 37 is along a horizontal plane, and the straight line along which the third conveying path 37 runs is parallel to the conveying direction 6. The third conveying path 37 includes a worker waste supply area 38 and an improperly sorted waste image capture area 39. The improperly sorted waste image capture area 39 is located downstream of the worker waste supply area 38 in the conveying direction 6. The third conveying device 33 conveys objects placed on the third conveying path 37 along the third conveying path 37 in the conveying direction 6. The third conveying device 33 is controlled by the control device 36 to ultimately place the objects placed on the third conveying path 37 in the first improperly sorted waste supply area 16 or the second improperly sorted waste supply area 24.

[0015] The label information detection camera 34 is disposed near the first worker waste removal area 15 and near the second worker waste removal area 23. The label information detection camera 34 is controlled by the control device 36 to capture an image showing the first worker waste removal area 15 and the second worker waste removal area 23. The annotation imaging unit 35 is disposed near the incorrectly sorted waste image capturing area 39. The annotation imaging unit 35 is controlled by the control device 36 to capture an image showing the incorrectly sorted waste image capturing area 39.

[0016] As shown in Figure 2, the object processing apparatus 1 further includes an inverting device 41. Figure 2 is a side view showing the inverting device 41. The inverting device 41 is formed from an articulated robot. The inverting device 41 is disposed near the incorrectly sorted waste image capturing area 39 on the third conveying path 37. Under the control of the control device 36, the inverting device 41 grasps objects placed in the incorrectly sorted waste image capturing area 39, inverts the grasped objects, and releases the grasped objects.

[0017] 3 is a block diagram showing the object processing apparatus 1. The control device 36 is a computer and includes a storage device 42 and a CPU (Central Processing Unit) 43. The storage device 42 records computer programs installed in the control device 36 and records information used by the CPU 43. The CPU 43 processes information and controls the storage device 42 by executing the computer programs installed in the control device 36. The CPU 43 also controls the sorting imaging unit 31, the robot 32, the third transport device 33, the label information detection camera 34, the annotation imaging unit 35, and the reversing device 41.

[0018] The computer programs installed in the control device 36 include a plurality of computer programs that respectively cause the control device 36 to realize a plurality of functions. The plurality of functions include an object recognition unit 44, an object selection unit 45, a training data creation unit 46, an object return unit 47, and a learning unit 48.

[0019] The object recognition unit 44 controls the sorting imaging unit 31 so that a waste sorting image is captured that includes multiple images of multiple objects placed in the sorting image capturing area 12 of the first transport path 5. The object recognition unit 44 processes the waste sorting image and, using a learning model created by machine learning, associates label information with the multiple objects appearing in the waste sorting image to calculate multiple sorting data. The object sorting unit 45 controls the robot 32 based on the multiple sorting data calculated by the object recognition unit 44 so that an object associated with recyclable waste to be processed, among the multiple objects placed in the robot waste removal area 14, is placed in the robot waste supply area 22.

[0020] The teacher data creation unit 46 controls the label information detection camera 34 to intermittently capture monitoring images showing the first worker waste removal area 15 and the second worker waste removal area 23. The teacher data creation unit 46 processes the monitoring images to determine whether target recyclable waste has been removed from the first worker waste removal area 15, and whether non-target waste has been removed from the second worker waste removal area 23. The teacher data creation unit 46 controls the annotation capture unit 35 to capture annotation images showing objects placed in the incorrectly sorted waste image capture area 39. When it is determined that target recyclable waste has been removed from the first worker waste removal area 15, the teacher data creation unit 46 creates teacher data that associates the captured annotation image with the target recyclable waste, and records the created teacher data in the storage device 42. When it is determined that non-target waste has been removed from the second worker waste removal area 23, the teacher data creation unit 46 creates teacher data that associates the captured annotation image with the non-target waste and records it in the memory device 42.

[0021] When it is determined that the target recyclable waste has been removed from the first worker waste removal area 15, the object return unit 47 controls the third conveying device 33 so that the target recyclable waste supplied to the third conveying path 37 is supplied to the first incorrectly separated waste supply area 16. When it is determined that the non-target waste has been removed from the second worker waste removal area 23, the object return unit 47 controls the third conveying device 33 so that the non-target waste supplied to the third conveying path 37 is supplied to the second incorrectly separated waste supply area 24. The learning unit 48 uses the training data created by the training data creation unit 46 to additionally train the learning model used by the object recognition unit 44, and updates the learning model used by the object recognition unit 44.

[0022] [Operation of automatic recyclable waste sorting device 10] The operation of the automatic recyclable waste sorting device 10 includes an operation for sorting multiple pieces of waste and an operation for updating a learning model. In the operation for sorting multiple pieces of waste, first, a user operates the first conveying device 2 and starts the first conveying device 2, and then operates the second conveying device 3 and starts the second conveying device 3. The user then places multiple pieces of waste in the waste supply area 11 of the first conveying path 5. The multiple pieces of waste include multiple pieces of recyclable waste A to be treated and multiple pieces of non-target waste B. The multiple pieces of recyclable waste A to be treated are recyclable waste that need to be removed from the first conveying path 5 and moved to a recyclable waste bin. An example of the multiple pieces of recyclable waste A to be treated is bottles made of glass colored a predetermined color (e.g., brown). The multiple pieces of non-target waste B are waste different from the multiple pieces of recyclable waste A to be treated and need to be placed in a non-target waste bin.

[0023] The plurality of pieces of trash placed in trash supply area 11 are transported by the first transport device in transport direction 6 along first transport path 5 and placed in sorting image capture area 12. The plurality of pieces of trash placed in sorting image capture area 12 are further transported in transport direction 6 along first transport path 5 and placed in robot trash removal area 14. The plurality of pieces of trash placed in robot trash removal area 14 are further transported in transport direction 6 along first transport path 5 and placed in a non-target trash disposal area.

[0024] The control device 36 controls the separation imaging unit 31 to capture a waste sorting image showing multiple pieces of waste placed in the separation image capturing area 12. The control device 36 uses a learning model created in advance by machine learning to determine the types of multiple pieces of waste and associates multiple pieces of label information with the multiple pieces of waste. The label information associated with a certain piece of waste among the multiple pieces of label information indicates the label "recyclable waste A to be treated" when the waste is recyclable waste A to be treated, and indicates the label "non-recyclable waste B" when the waste is different from recyclable waste A to be treated.

[0025] The control device 36 controls the robot 32 to remove the recyclable waste A to be treated that is associated with the label "recyclable waste A to be treated" from among the multiple pieces of waste placed in the robot waste removal area 14, from the first conveying path 5. The control device 36 further controls the robot 32 to place the recyclable waste A to be treated that has been removed from the first conveying path 5 in the robot waste supply area 22 of the second conveying path 21.

[0026] The plurality of pieces of trash placed in the robot trash supply area 22 are transported by the second transport device 3 along the second transport path 21 in the transport direction 6 and placed in the second worker trash removal area 23. The trash placed in the second worker trash removal area 23 is further transported along the second transport path 21 in the transport direction 6 and placed in the target recyclable trash storage area.

[0027] In the automatic recyclable waste sorting device 10, the control device 36 may erroneously associate the target recyclable waste A with the label "non-target recyclable waste B," thereby not removing the target recyclable waste A from the first conveying path 5. Furthermore, in the automatic recyclable waste sorting device 10, the control device 36 may erroneously associate the non-target recyclable waste B with the label "target recyclable waste A," thereby placing the non-target recyclable waste B in the robot waste supply area 22 of the second conveying path 21. For this reason, as shown in FIG. 4 , while the operation of sorting the multiple pieces of waste is being performed, the worker 51 monitors the multiple pieces of waste placed in the first worker waste removal area 15 and the second worker waste removal area 23. FIG. 4 is a plan view showing the worker 51 monitoring the multiple pieces of waste placed in the first worker waste removal area 15 and the second worker waste removal area 23.

[0028] When non-target waste B is placed in the second worker waste removal area 23, the worker 51 removes the non-target waste B from the second conveying path 21 in the second worker waste removal area 23. The worker 51 then places the removed non-target waste B in the worker waste supply area 38, as shown in Figure 5. Figure 5 is a plan view showing the operation of the worker 51 placing the non-target waste B in the worker waste supply area 38.

[0029] When recyclable waste A to be processed is placed in the first worker waste removal area 15, the worker 51 removes the recyclable waste A from the first conveying path 5 in the first worker waste removal area 15, as shown in FIG. 6. FIG. 6 is a plan view showing the action of the worker 51 removing the recyclable waste A to be processed from the first conveying path 5 in the first worker waste removal area 15. The worker 51 places the recyclable waste A to be processed that has been removed from the first conveying path 5 in the worker waste supply area 38, similar to the non-target waste B that has been removed from the second conveying path 21. The waste placed in the worker waste supply area 38 is transported by the third conveying device 33 along the third conveying path 37 in the conveying direction 6 and placed in the incorrectly sorted waste image capture area 39.

[0030] 7 is a flowchart showing part of the operation for updating the learning model. While the operation of separating multiple types of waste is being performed, the control device 36 controls the label information detection camera 34 to intermittently capture monitoring images showing the first worker waste removal area 15 and the second worker waste removal area 23. The control device 36 processes the monitoring images to determine whether the worker 51 has removed non-target waste B from the first worker waste removal area 15, and whether the worker 51 has removed target recyclable waste A from the second worker waste removal area 23.

[0031] While the operation of separating the multiple pieces of waste is being performed, the control device 36 further controls the annotation imaging unit 35 to intermittently capture an improperly separated waste imaging area monitoring image that captures the improperly separated waste image imaging area 39. When the control device 36 determines that the recyclable waste A to be treated has been removed from the first worker waste removal area 15 (step S1, Yes), the control device 36 performs image processing on the improperly separated waste imaging area monitoring image and determines whether the recyclable waste A to be treated has been placed in the improperly separated waste image imaging area 39. When the control device 36 determines that the recyclable waste A to be treated has been placed in the improperly separated waste image imaging area 39, the control device 36 controls the annotation imaging unit 35 to capture a first recyclable waste image that captures the recyclable waste A to be treated (step S2).

[0032] After the first image of recyclable waste to be treated is captured, the recyclable waste A is transported by the third transport device 33 along the third transport path 37 in the transport direction 6 and is positioned downstream of the incorrectly separated waste image capturing area 39 in the transport direction 6. After the first image of recyclable waste to be treated is captured, the control device 36 controls the inverting device 41 to grasp the recyclable waste 52 to be treated that is positioned downstream of the incorrectly separated waste image capturing area 39 in the transport direction 6, as shown in FIG. 8. FIG. 8 is a side view showing the recyclable waste 52 grasped by the inverting device 41. After the recyclable waste 52 to be treated is grasped, the control device 36 controls the inverting device 41 to invert the recyclable waste 52 to be treated, as shown in FIG. 9. FIG. 9 is a side view showing the recyclable waste 52 to be treated that has been inverted by the inverting device 41. After the recyclable waste 52 to be processed has been turned over, the control device 36 controls the turnover device 41 to place the recyclable waste 52 to be processed in the incorrectly sorted waste image capturing area 39, as shown in Fig. 10. Fig. 10 is a side view showing the recyclable waste 52 to be processed that has been placed in the incorrectly sorted waste image capturing area 39 by the turnover device 41.

[0033] After the inverted recyclable waste A to be processed is placed in the incorrectly separated waste image capturing area 39, the control device 36 controls the annotation capturing unit 35 to capture a second image of recyclable waste to be processed, which captures the recyclable waste A to be processed. The second image of recyclable waste to be processed differs from the first image of recyclable waste to be processed because the recyclable waste A to be processed is inverted. The control device 36 records the first image of recyclable waste to be processed in the storage device 42 in association with the label "recyclable waste A to be processed," and records the second image of recyclable waste to be processed in the storage device 42 in association with the label "recyclable waste A to be processed" (step S3).

[0034] After the first image of recyclable waste to be treated and the second image of recyclable waste to be treated are captured, the control device 36 controls the third conveying device 33 to place the recyclable waste A to be treated that is placed in the incorrectly separated waste image capturing area 39 in the second incorrectly separated waste supply area 24 (step S4). The recyclable waste A to be treated that is placed in the second incorrectly separated waste supply area 24 is conveyed in the conveying direction 6 along the second conveying path 21 and placed in the recyclable waste storage area.

[0035] When it is determined that non-target waste B has been removed from the second worker waste removal area 23 (Step S5, Yes), the control device 36 performs image processing on the mis-separated waste imaging area monitoring image to determine whether non-target waste B has been placed in the mis-separated waste image imaging area 39. When it is determined that non-target waste B has been placed in the mis-separated waste image imaging area 39, the control device 36 controls the annotation imaging unit 35 and the reversing device 41 to capture a first non-target waste image and a second non-target waste image in which the non-target waste B appears (Step S6). The first non-target waste image and the second non-target waste image, like the first treatment-target recyclable waste image and the second treatment-target recyclable waste image, respectively capture non-target waste B as viewed from two different directions.

[0036] The control device 36 associates the first non-target waste image with the label "non-target waste B" and records it in the storage device 42, and associates the second non-target waste image with the label "non-target waste B" and records it in the storage device 42 (step S7). After the first non-target waste image and the second non-target waste image have been captured, the control device 36 controls the third conveying device 33 to place the non-target waste B placed in the incorrectly separated waste image capturing area 39 in the first incorrectly separated waste supply area 16 of the first conveying path 5 (step S8). The non-target waste B placed in the first incorrectly separated waste supply area 16 is conveyed along the first conveying path 5 in the conveying direction 6 and placed in the non-target waste disposal area.

[0037] The processing of steps S1 to S8 is repeatedly executed until a predetermined number of first recyclable waste images, second recyclable waste images, first non-target waste images, and second non-target waste images are recorded in the storage device 42. When a predetermined number of first recyclable waste images, second recyclable waste images, first non-target waste images, and second non-target waste images are recorded in the storage device 42, the control device 36 creates multiple sets of training data. The multiple sets of training data associate multiple images with multiple sets of label information. Each of the multiple images is one of the first recyclable waste image, the second recyclable waste image, the first non-target waste image, and the second non-target waste image. Of the multiple sets of label information, the label information associated with the first recyclable waste image or the second recyclable waste image indicates the label "target recyclable waste A." Of the multiple sets of label information, the label information associated with the first non-target waste image or the second non-target waste image indicates the label "non-target waste B."

[0038] The control device 36 uses multiple training data sets to additionally train a learning model that associates label information with multiple pieces of garbage appearing in the garbage sorting image, thereby updating the learning model. By updating the learning model in this manner, the object processing device 1 can improve the accuracy of associating multiple objects appearing in the garbage sorting image with label information. For example, the object processing device 1 can reduce the frequency with which target recyclable garbage similar to target recyclable garbage incorrectly associated with the label "non-target garbage B" is mistakenly associated with the label "non-target garbage B." The object processing device 1 can also reduce the frequency with which non-target garbage similar to target recyclable garbage incorrectly associated with the label "target recyclable garbage A" is mistakenly associated with the label "target recyclable garbage A." Furthermore, when creating multiple training data sets for additional training of the learning model, the object processing device 1 eliminates the need for the worker 51 to perform additional tasks separate from the task of correctly re-sorting incorrectly sorted garbage, thereby reducing the burden on the worker 51.

[0039] [Effects of the object processing device 1 of the first embodiment] The object processing apparatus 1 of the first embodiment includes a first conveying device 2, a sorting imaging unit 31, a robot 32, a second conveying device 3, an annotation imaging unit 35, a label information detection camera 34, and a learning unit 48. The first conveying device 2 conveys multiple pieces of waste along a first conveying path 5. The sorting imaging unit 31 captures a waste sorting image that shows the first conveying path 5. The robot 32 uses a learning model created by machine learning to move recyclable waste A to be processed that has been sorted from the multiple pieces of waste based on the waste sorting image from the first conveying path 5 to the second conveying path 21. The second conveying device 3 conveys the recyclable waste A to be processed along the second conveying path 21. The annotation imaging unit 35 captures a first recyclable waste image to be processed or a first non-target waste image that shows improperly sorted waste that has been moved from the first conveying path 5 or the second conveying path 21 to the improperly sorted waste image capturing area 39. The label information detection camera 34 captures monitoring images for automatically determining label information to be assigned to incorrectly sorted waste. The learning unit 48 updates the learning model based on training data that associates the first target recyclable waste image or the first non-target waste image with label information.

[0040] In this case, the object processing apparatus 1 of Example 1 can automatically create training data for training the learning model by having the operator 51 move the incorrectly sorted waste from the first conveying path 5 or the second conveying path 21 to the incorrectly sorted waste image capturing area 39. The object processing apparatus 1 of Example 1 can improve the accuracy of associating multiple objects shown in the waste sorting image with label information by updating the learning model using the training data. When creating training data, the object processing apparatus 1 of Example 1 does not require the operator 51 to perform an additional task separate from the task of correctly re-sorting the incorrectly sorted waste, thereby reducing the burden on the operator 51.

[0041] Furthermore, the control device 36 of the object processing device 1 of Example 1 generates label information by determining whether the incorrectly sorted waste has been removed from the first conveying path 5 or the second conveying path 21 based on the monitoring image captured by the label information detection camera 34. In this case, the object processing device 1 of Example 1 does not require the worker 51 to perform the additional task of identifying label information to be associated with the first target recyclable waste image or the first non-target waste image, further reducing the burden on the worker 51 when creating training data.

[0042] Furthermore, the object processing apparatus 1 of Example 1 further includes a third conveying device 33 that supplies the incorrectly sorted waste to a conveying path selected based on the label information from the first conveying path 5 and the second conveying path 21 after the first image of recyclable waste to be processed or the first image of non-target waste has been captured. In this case, the object processing apparatus 1 of Example 1 does not require the worker 51 to return the incorrectly sorted waste to the first conveying path 5 or the second conveying path 21 after the first image of recyclable waste to be processed or the first image of non-target waste has been captured, further reducing the burden on the worker 51.

[0043] The object processing device 1 of Example 1 also includes an inverting device 41 that changes the orientation of the missorted waste in the missorted waste image capturing area 39. After the orientation of the missorted waste is changed, the annotation capturing unit 35 captures a second image of recyclable waste to be processed or a second image of non-target waste that captures the missorted waste. The learning unit 48 updates the learning model based on the second image of recyclable waste to be processed or the second image of non-target waste. In this case, the object processing device 1 of Example 1 can additionally train the learning model using additional training data showing images of the missorted waste viewed from other directions, thereby further improving the accuracy of associating multiple wastes with label information. The object processing device 1 of Example 1 does not require the operator 51 to perform additional work to create additional training data showing images of the missorted waste viewed from other directions, thereby reducing the burden on the operator 51 when creating training data. [Example]

[0044] As shown in Fig. 11, the object processing apparatus of Example 2 is the same as the object processing apparatus 1 of Example 1 described above, except that the label information detection camera 34 of the object processing apparatus 1 of Example 1 described above is replaced with an input device 61. Fig. 11 is a plan view showing the object processing apparatus of Example 2. The input device 61 has a target recyclable waste A button 62 and a non-target recyclable waste B button 63. The input device 61 is disposed near the worker waste supply area 38. The input device 61 outputs to the control device 36 whether the target recyclable waste A button 62 has been pressed, and outputs to the control device 36 whether the non-target recyclable waste B button 63 has been pressed.

[0045] When non-target waste B is placed on the second conveying path 21, the worker 51 removes the non-target waste B from the second conveying path 21 and places it in the worker waste supply area 38, and presses the non-target waste B button 63. When recyclable waste A to be processed is placed on the first conveying path 5, the worker 51 removes the recyclable waste A to be processed from the first conveying path 5 and places it in the worker waste supply area 38, and presses the recyclable waste A to be processed button 62.

[0046] When the non-target trash B button 63 is pressed, the control device 36 controls the annotation imaging unit 35 and the reversing device 41 to capture a first non-target trash image and a second non-target trash image, each showing the non-target trash B from two different directions. The control device 36 then associates the first non-target trash image and the second non-target trash image with the label "non-target trash B" and records them in the storage device 42. After the first non-target trash image and the second non-target trash image are captured, the control device 36 controls the third conveying device 33 to place the non-target trash B located in the incorrectly separated trash image capturing area 39 in the first incorrectly separated trash supply area 16 of the first conveying path 5. The non-target trash B placed in the first incorrectly separated trash supply area 16 is then conveyed along the first conveying path 5 in the conveying direction 6 and placed in the non-target trash disposal area.

[0047] When the target recyclable waste A button 62 is pressed, the control device 36 controls the annotation imaging unit 35 and the reversing device 41 to capture a first target recyclable waste image and a second target recyclable waste image, each showing the target recyclable waste A from two different directions. The control device 36 then associates the first target recyclable waste image and the second target recyclable waste image with the label "target recyclable waste A" and records them in the storage device 42. After the first target recyclable waste image and the second target recyclable waste image have been captured, the control device 36 controls the third conveying device 33 to place the target recyclable waste A located in the incorrectly separated waste image capturing area 39 in the second incorrectly separated waste supply area 24 of the second conveying path 21. The target recyclable waste A placed in the second incorrectly separated waste supply area 24 is then conveyed along the second conveying path 21 in the conveying direction 6 and placed in the target recyclable waste storage area.

[0048] The object processing device of Example 2, like the object processing device 1 of Example 1 described above, creates training data and uses the training data to perform additional training on a learning model and update it. Therefore, like the object processing device 1 of Example 1 described above, the object processing device of Example 2 can improve the accuracy of associating multiple objects shown in a waste sorting image with label information. Furthermore, when identifying label information to associate with incorrectly sorted waste, the worker 51 simply presses the target recyclable waste A button 62 or the non-target waste B button 63. Therefore, the object processing device of Example 2 can reduce the burden on the worker 51 when creating training data. [Example]

[0049] As shown in FIG. 12 , the object processing apparatus of Example 3 is identical to the object processing apparatus 1 of Example 1 described above, except that the third conveying device 33 of the object processing apparatus 1 of Example 1 described above is replaced with a fourth conveying device 71 and a fifth conveying device 72. FIG. 12 is a plan view showing the object processing apparatus of Example 3. A fourth conveying path 73 is formed in the fourth conveying device 71. The fourth conveying path 73 extends along a horizontal plane, and the straight line along which the fourth conveying path 73 extends is parallel to the conveying direction 6. The fourth conveying path 73 includes a non-target waste supply area 74 and a non-target waste image capturing area 75. The non-target waste image capturing area 75 is located downstream of the non-target waste supply area 74 in the conveying direction 6. The fourth conveying device 71 conveys objects placed on the fourth conveying path 73 along the fourth conveying path 73 in the conveying direction 6. The fourth conveying device 71 ultimately places the objects placed on the fourth conveying path 73 in the first missorted waste supply area 16.

[0050] The fifth conveying device 72 is formed with a fifth conveying path 76. The fifth conveying path 76 extends along a horizontal plane, and the straight line along which the fifth conveying path 76 extends is parallel to the conveying direction 6. The fifth conveying path 76 includes a recyclable waste supply area 77 to be processed and a recyclable waste image capture area 78 to be processed. The recyclable waste image capture area 78 to be processed is located downstream of the recyclable waste supply area 77 to be processed in the conveying direction 6. The fifth conveying device 72 conveys objects placed on the fifth conveying path 76 along the fifth conveying path 76 in the conveying direction 6. The fifth conveying device 72 ultimately places the objects placed on the fifth conveying path 76 in the second missorted waste supply area 24.

[0051] When non-target waste B is placed on the second conveying path 21, the worker 51 removes the non-target waste B from the second conveying path 21 and places it in the non-target waste supply area 74. The fourth conveying device 71 conveys the non-target waste B placed in the non-target waste supply area 74 along the fourth conveying path 73 in the conveying direction 6, places it in the non-target waste image capturing area 75, and places it in the first misseparated waste supply area 16. When recyclable waste A to be treated is placed on the first conveying path 5, the worker 51 removes the recyclable waste A to be treated from the first conveying path 5 and places it in the recyclable waste supply area 77 to be treated. The fifth conveying device 72 conveys the recyclable waste A placed in the recyclable waste supply area 77 to be treated along the fifth conveying path 76 in the conveying direction 6, places it in the recyclable waste image capturing area 78 to be treated, and places it in the second misseparated waste supply area 24.

[0052] The control device 36 performs image processing on the misseparated waste imaging area monitoring image, determines whether non-target waste B has been placed in the non-target waste image imaging area 75, and determines whether target recyclable waste A has been placed in the target recyclable waste image imaging area 78. When it is determined that non-target waste B has been placed in the non-target waste image imaging area 75, the control device 36 captures a first non-target waste image and a second non-target waste image showing the non-target waste B viewed from two different directions, similar to the object processing device 1 of the first embodiment described above. When it is determined that target recyclable waste A has been placed in the target recyclable waste image imaging area 78, the control device 36 captures a first target recyclable waste image and a second target recyclable waste image showing the target recyclable waste A viewed from two different directions.

[0053] The object processing device of Example 3, like the object processing device 1 of Example 1 described above, creates training data and uses the training data to perform additional training on a learning model to update it. Therefore, like the object processing device 1 of Example 1 described above, the object processing device of Example 3 can improve the accuracy of associating multiple objects shown in a waste sorting image with label information. Furthermore, when identifying label information to associate with incorrectly sorted waste, the worker 51 simply places the incorrectly sorted waste in the non-target waste image capture area 75 or the target recyclable waste image capture area 78. Therefore, the object processing device of Example 3 can reduce the burden on the worker 51 when creating training data. [Example]

[0054] As shown in Fig. 13, the object processing apparatus of Example 4 is the same as the object processing apparatus of Example 2 except that the third transport device 33 of the object processing apparatus of Example 2 described above is replaced with a mounting table 81. Fig. 13 is a plan view showing the object processing apparatus of Example 4. The annotation imaging unit 35 of the object processing apparatus of Example 4 captures an image showing an object placed on the mounting table 81.

[0055] When non-target waste B is placed on the second conveyance path 21, the worker 51 removes the non-target waste B from the second conveyance path 21 and places it on the placement table 81, and presses the non-target waste B button 63. When the non-target waste B button 63 is pressed, the control device 36 controls the annotation imaging unit 35 to capture a non-target waste image showing the non-target waste B, and associates the non-target waste image with the label "Non-target waste B" and records it in the storage device 42. After the non-target waste image has been captured, the worker 51 places the non-target waste B that was placed on the placement table 81 on the first conveyance path 5.

[0056] When recyclable waste A to be processed is placed on the first transport path 5, the worker 51 removes the recyclable waste A to be processed from the first transport path 5 and places it on the placement table 81, and presses the recyclable waste A to be processed button 62. When the recyclable waste A to be processed button 62 is pressed, the control device 36 controls the annotation imaging unit 35 to capture an image of the recyclable waste A to be processed, and associates the image of the recyclable waste with the label "recyclable waste A to be processed" and records it in the storage device 42. After the image of the recyclable waste to be processed has been captured, the worker 51 places the recyclable waste A to be processed that has been placed on the placement table 81 on the second transport path 21.

[0057] The object processing device of Example 4, like the object processing device 1 of Example 1 described above, creates training data and uses the training data to perform additional training on a learning model and update it. Therefore, like the object processing device 1 of Example 1 described above, the object processing device of Example 4 can improve the accuracy of associating multiple objects shown in a waste sorting image with label information. Furthermore, when identifying label information to associate with incorrectly sorted waste, the worker 51 simply presses the target recyclable waste A button 62 or the non-target waste B button 63. Therefore, the object processing device of Example 4 can reduce the burden on the worker 51 when creating training data. [Example]

[0058] As shown in Fig. 14, in the object processing apparatus of Example 5, the missorted waste image capturing area 39 of the third conveying device 33 of the object processing apparatus 1 of Example 1 described above is replaced with another missorted waste image capturing area 91. Fig. 14 is a side view showing the missorted waste image capturing area 91 of the object processing apparatus of Example 5. The missorted waste image capturing area 91 is inclined with respect to a horizontal plane 90 so that the area of ​​the missorted waste image capturing area 91 downstream in the conveying direction 6 is lower than the area of ​​the missorted waste image capturing area 91 upstream in the conveying direction 6.

[0059] When non-target waste B is placed in the first worker waste removal area 15, the worker 51 removes the non-target waste B from the second conveying path 21 and places it in the worker waste supply area 38. When recyclable waste A to be processed is placed in the second worker waste removal area 23, the worker 51 removes the recyclable waste A to be processed from the first conveying path 5 and places it in the worker waste supply area 38. The incorrectly separated waste 92 placed in the worker waste supply area 38 is transported by the third conveying device 33 along the third conveying path 37 in the transport direction 6 and placed in the incorrectly separated waste image capture area 91. When the control device 36 determines based on the incorrectly separated waste image capture area monitoring image that the incorrectly separated waste 93 has been placed in the incorrectly separated waste image capture area 91, the control device 36 controls the annotation capture unit 35 to capture a first incorrectly separated waste image showing the incorrectly separated waste 93.

[0060] The missorted waste 93 placed in the missorted waste image capturing area 91 rolls in the conveying direction 6 because the missorted waste image capturing area 91 is tilted. The rolled missorted waste 94 is placed in an area of ​​the missorted waste image capturing area 91 downstream in the conveying direction 6 from the area where the missorted waste 93 was placed at a second timing, which is a predetermined time after the first timing at which the first missorted waste image was captured. Furthermore, the part of the missorted waste 94 facing the annotation capturing unit 35 is different from the part of the missorted waste 93 facing the annotation capturing unit 35. The control device 36 controls the annotation capturing unit 35 at the second timing to capture a second missorted waste image showing the missorted waste 94.

[0061] Similar to the object processing apparatus 1 of the first embodiment, when the control device 36 determines based on the monitoring image that non-target waste B has been removed from the second conveying path 21, the control device 36 associates the first incorrectly separated waste image and the second incorrectly separated waste image with “non-target waste B” and records them in the storage device 42. When the control device 36 determines based on the monitoring image that non-target waste B has been removed from the second conveying path 21, the control device 36 further controls the third conveying device 33 to place the incorrectly separated waste in the first incorrectly separated waste supply area 16. When the control device 36 determines based on the monitoring image that target recyclable waste A has been removed from the first conveying path 5, the control device 36 associates the first incorrectly separated waste image and the second incorrectly separated waste image with “target recyclable waste A” and records them in the storage device 42. When the control device 36 determines based on the monitoring image that target recyclable waste A has been removed from the first conveying path 5, the control device 36 further controls the third conveying device 33 to place the incorrectly separated waste in the second incorrectly separated waste supply area 24.

[0062] The object processing device of Example 5, like the object processing device 1 of Example 1 described above, creates training data and uses the training data to perform additional training and update the learning model. Therefore, like the object processing device 1 of Example 1 described above, the object processing device of Example 5 can improve the accuracy of associating multiple objects shown in a waste sorting image with label information. The object processing device of Example 4 does not use the reversing device 41, and captures a first incorrectly sorted waste image and a second incorrectly sorted waste image, each of which shows incorrectly sorted waste viewed from two different directions. Therefore, the object processing device of Example 4 can reduce the burden on the control device 36 compared to the object processing device 1 of Example 1.

[0063] In the object processing device of the embodiment described above, two images of misclassified waste are captured, each showing the misclassified waste viewed from two different directions, but more than three images of misclassified waste may be captured, or only one image of misclassified waste may be captured. Even in such cases, the object processing device of the embodiment can improve the accuracy of associating multiple pieces of waste with label information, and can reduce the burden on worker 51 when creating training data.

[0064] Incidentally, the object processing apparatus in the above-described embodiment separates a plurality of types of waste, but it may also separate a plurality of objects other than waste.

[0065] Although the embodiments have been described above, the embodiments are not limited to the above content. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be combined as appropriate. Furthermore, at least one of various omissions, substitutions, and modifications of the components can be made without departing from the spirit of the embodiments. [Explanation of symbols]

[0066] 10: Automatic recyclable waste sorting device 1: Material processing equipment 2: First conveying device 3: Second conveying device 5: First transport path 15: First worker garbage removal area 21: Second transport path 23: Second worker garbage removal area 31: Imaging unit for classification 32:Robot 33: Third transport device 34: Label information detection camera 35: Imaging unit for annotation 37: Third transport route 39: Missorted garbage image capture area 41: Inverter 48: Learning Department 51: Worker 61: Input device 71: 4th transport device 72: 5th transport device 75: Non-target garbage image capturing area 78: Image capture area of ​​recyclable waste to be processed 90: Horizontal plane 91: Missorted garbage image capture area

Claims

1. a first conveying unit that conveys a plurality of objects along a first conveying path; a first imaging unit that captures a first image in which the first transport path is captured; a robot that moves a first object selected from the plurality of objects based on the first image using a learning model from the first transport path to a second transport path; a second conveying unit that conveys the first object along the second conveying path; a second imaging unit configured to capture a second image of a second object among the plurality of objects that has been moved from the first transport path or the second transport path to an imaging area; a label information generation unit that generates label information to be assigned to the second object; a learning unit that updates the learning model based on the second image and the label information; An object processing device comprising:

2. The label information generating unit generates the label information in response to a user operation. The object processing device of claim 1 .

3. The label information generating unit generates the label information by detecting whether the second object has been moved from the first transport path or the second transport path. The object processing device of claim 1 .

4. a third conveying section that supplies the second object to a conveying path selected based on the label information from the first conveying path and the second conveying path after the second image is captured; The object processing device of claim 1 further comprising:

5. a fourth conveying unit that moves the object that has been moved to a first image capturing area of ​​the image capturing area to the first conveying path; a fifth conveying unit that moves the object that has been moved to a second image capturing area of ​​the image capturing area to the second conveying path, The label information generating unit generates the label information by determining whether the second object has been moved to the first image capturing area or the second image capturing area based on the second image. The object processing device of claim 1 .

6. a mechanism for changing the orientation of the second object in the imaging region; the second imaging unit further captures a third image in which the second object is captured after the orientation is changed; The learning unit updates the learning model further based on the third image. The object processing device of claim 1 .

7. a placement surface on which the second object is placed in the imaging area is along a plane inclined with respect to a horizontal plane; the second imaging unit further captures a third image in which the second object appears at a timing different from the timing at which the second image is captured; The learning unit updates the learning model further based on the third image. The object processing device of claim 1 .

Images