Object Sorting System

Abstract

An object sorting system, wherein: a camera 20 captures a first image of an object group in a transport path along which the object group is transported; an image recognition unit 11 recognizes a second image, which is an image of objects present in the first image, by using a trained model, assigns feature information indicating a feature of the second image to the second image, generates a recognition result in which the feature information for second image is assigned to the first image, and stores the recognition result in a recognition result storage unit 12; an object sorting device 30a sorts a desired object, on the basis of the recognition result, from within the object group being transported along the transport path; and, when an undesired object different than the desired object is erroneously sorted by the object sorting device 30a, a recognition result extraction unit 13a assesses that the recognition result stored in the recognition result storage unit 12 is an erroneous recognition result including an image of the undesired object that was erroneously sorted and extracts the erroneous recognition result from the recognition result storage unit 12.

Description

[Technical Field]

[0001] The present disclosure relates to an object sorting system. [Background technology]

[0002] At waste disposal sites, large amounts of waste are processed daily on conveyor belts. At the waste disposal sites, the waste is sorted by hand. While sorting waste is a simple task, it places a heavy burden on the workers who sort the waste (hereinafter referred to as "sorters"). Therefore, devices that automatically sort waste (hereinafter referred to as "waste sorting devices") have been developed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6854995 specification Summary of the Invention [Problem to be solved by the invention]

[0004] If a waste sorting device were to take over the tasks previously performed by sorters, the waste sorting device could recognize each piece of waste flowing on the belt conveyor and, based on the recognition results, use a robotic hand or suction pad to extract desired waste (hereinafter sometimes referred to as "desired waste") from the mass of waste flowing on the belt conveyor. Therefore, when multiple types of waste are mixed and flowing on the belt conveyor, the waste sorting device needs to identify the type of waste. For the waste sorting device to recognize various types of waste, recognition using a trained model generated by machine learning is effective.

[0005] However, machine learning requires a huge amount of training data, and when generating training data from image data of various types of waste, if the annotation work on the image data is done manually by visually inspecting the images, generating the training data will require a huge amount of effort.

[0006] Therefore, the present disclosure proposes a technology that can reduce the effort required to generate training data. [Means for solving the problem]

[0007] The object sorting system disclosed herein includes a camera, a recognition unit, a memory unit, a sorting unit, and an extraction unit. The camera captures a first image of a group of objects on a transport path along which the group of objects is transported. The recognition unit uses a trained model to recognize a second image of each object present in the first image, assigns feature information to the second image that indicates features of the second image, and generates a recognition result in which the feature information is assigned to the second image in the first image. The memory unit stores the recognition result. The sorting unit sorts a desired object from the group of objects transported on the transport path based on the recognition result. When the sorting unit erroneously selects an undesired object other than the desired object, the extraction unit determines an erroneous recognition result in the recognition result stored in the memory unit, which is a recognition result that includes an image of the undesired object that was erroneously selected, and extracts the erroneous recognition result from the memory unit. [Effects of the Invention]

[0008] The disclosed technology can reduce the effort required to generate training data. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration example of an object sorting system according to a first embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating a configuration example of the control device according to the first embodiment of the present disclosure. [Figure 3]FIG. 3 is a diagram illustrating an example of feature information according to the first embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example of tracing the centroid coordinates according to the first embodiment of the present disclosure. [Figure 5] FIG. 5 is a diagram illustrating an example of tracing the barycentric coordinates according to the first embodiment of the present disclosure. [Figure 6] FIG. 6 is a diagram illustrating an example of tracing the centroid coordinates according to the first embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating an example of tracing the centroid coordinates according to the first embodiment of the present disclosure. [Figure 8] FIG. 8 is a diagram showing an example of extraction of desired waste in the first embodiment of the present disclosure. [Figure 9] FIG. 9 is a diagram illustrating an example of the operation of the control device according to the first embodiment of the present disclosure. [Figure 10] FIG. 10 is a diagram illustrating an example of the operation of the control device according to the first embodiment of the present disclosure. [Figure 11] FIG. 11 is a diagram illustrating an example of the operation of the control device according to the first embodiment of the present disclosure. [Figure 12] FIG. 12 is a diagram illustrating an example of the operation of the control device according to the first embodiment of the present disclosure. [Figure 13] FIG. 13 is a diagram illustrating an example of the operation of the control device according to the first embodiment of the present disclosure. [Figure 14] FIG. 14 is a diagram illustrating an example of the operation of the control device according to the first embodiment of the present disclosure. [Figure 15] FIG. 15 is a diagram illustrating an example of the operation of the control device according to the first embodiment of the present disclosure. [Figure 16] FIG. 16 is a diagram illustrating an example of transition of button colors according to the first embodiment of the present disclosure. [Figure 17] FIG. 17 is a diagram illustrating an example of transition of button colors according to the first embodiment of the present disclosure. [Figure 18] FIG. 18 is a diagram illustrating a configuration example of a control device according to a second embodiment of the present disclosure. [Figure 19] FIG. 19 is a diagram illustrating a configuration example of a control device according to a third embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals.

[0011] [Example 1] <Configuration of object sorting system> FIG. 1 is a diagram illustrating a configuration example of an object sorting system according to a first embodiment of the present disclosure.

[0012] 1, the object sorting system 1 includes a control device 10, a camera 20, an object sorting device 30, a belt conveyor 40, a push button 50, a display 60, and an input device 70. The push button 50, the display 60, and the input device 70 are connected to the control device 10. Examples of the input device 70 include a pointing device such as a mouse, and a keyboard. The control device 10, the camera 20, and the object sorting device 30 are connected to each other via a network.

[0013] The following description will be given taking as an example a case where the object sorting system 1 shown in FIG. 1 is installed at a waste disposal site where waste materials flow on a belt conveyor 40. That is, the following description will be given taking as an example a case where the objects to be sorted by the object sorting system 1 are waste materials. However, the object sorting system 1 may also be installed in an assembly plant or the like where parts flow on a belt conveyor. That is, the objects to be sorted by the object sorting system 1 are not limited to waste materials, and the object sorting system 1 can be used for a variety of objects.

[0014] The belt conveyor 40 transports the waste mass placed on the belt conveyor 40 in the transport direction CD. In other words, the belt conveyor 40 forms a transport path along which the waste mass is transported in the transport direction CD.

[0015] The camera 20 is disposed above the belt conveyor 40 along which the waste mass is transported, has a predetermined angle of view, and continuously captures a predetermined area on the upper surface of the belt conveyor 40 from above the belt conveyor 40 at a constant frame rate. Therefore, the image captured by the camera 20 (hereinafter sometimes referred to as the "captured image") is an image of the waste mass. The captured image is transmitted from the camera 20 to the control device 10.

[0016] <Control device configuration> Fig. 2 is a diagram illustrating a configuration example of a control device according to a first embodiment of the present disclosure. A control device 10a illustrated in Fig. 2 corresponds to the control device 10 illustrated in Fig. 1, and an object sorting device 30a illustrated in Fig. 2 corresponds to the object sorting device 30 illustrated in Fig. 1. In Fig. 2, the control device 10a includes an image recognition unit 11, a recognition result storage unit 12, a recognition result extraction unit 13a, a label correction unit 14, a training data storage unit 15, a machine learning unit 16, a trained model storage unit 17, and a button color control unit 18.

[0017] The image recognition unit 11 uses a trained model stored in the trained model storage unit 17 to recognize each waste image (hereinafter sometimes referred to as a "waste image") present in the captured image, and assigns information indicating the characteristics of each waste image (hereinafter sometimes referred to as "feature information") to each recognized waste image. The image recognition unit 11 assigns feature information to waste images whose image recognition score is equal to or greater than a threshold TH1. The feature information includes information indicating the type of waste image (hereinafter sometimes referred to as "type information"), information indicating the contour of the waste image (hereinafter sometimes referred to as "contour information"), and coordinates of the area center of gravity of the waste image (hereinafter sometimes referred to as "center of gravity coordinates"). The image recognition unit 11 recognizes waste images, for example, by performing instance segmentation on the captured image. The image recognition unit 11 associates the waste images with the feature information already assigned to the waste images, generates a recognition result (hereinafter sometimes referred to as an "image recognition result") in which feature information is assigned to each waste image in the captured image, and outputs the generated image recognition result to the recognition result storage unit 12 and the object sorting device 30a. The recognition result storage unit 12 stores the image recognition results sequentially output from the image recognition unit 11 in chronological order. In other words, the recognition result storage unit 12 stores the captured images in which the waste images and feature information are associated with each other. In addition, the image recognition unit 11 outputs a signal (hereinafter sometimes referred to as a "recognition completion signal") indicating that the recognition of the waste images and the assignment of feature information to the waste images have been completed to the button color control unit 18.

[0018] The object sorting device 30a sorts desired waste by extracting it from the waste mass being transported on the belt conveyor 40 based on the image recognition results (i.e., waste images and characteristic information) output from the image recognition unit 11. The object sorting device 30a extracts the desired waste using, for example, a robot hand, a suction pad, or the like.

[0019] When the push button 50 is pressed, the recognition result extraction unit 13a determines, from the captured images stored in the recognition result storage unit 12, images (hereinafter sometimes referred to as "images containing incorrectly sorted waste") that are waste other than desired waste (hereinafter sometimes referred to as "undesired waste") and that include images of undesired waste that have been incorrectly sorted by the object sorting device 30a (hereinafter sometimes referred to as "images containing incorrectly sorted waste"). The recognition result extraction unit 13a also determines that a recognition result that includes an image containing incorrectly sorted waste is an incorrect recognition result. The recognition result extraction unit 13a then extracts, from the recognition result storage unit 12, the images containing incorrectly sorted waste and the feature information assigned to each waste image in the images containing incorrectly sorted waste as incorrect recognition results. The recognition result extraction unit 13a outputs the incorrect recognition results extracted from the recognition result storage unit 12 to the label correction unit 14 and displays them on the display 60. Furthermore, the recognition result extraction unit 13a outputs a signal indicating that extraction of the erroneous recognition results from the recognition result storage unit 12 has been completed (hereinafter, this may be referred to as an “extraction completion signal”) to the button color control unit 18.

[0020] The push button 50 is pressed by an operator, and the operator presses the push button 50 when undesired waste has been missorted by the object sorting device 30a. When the push button 50 is pressed, a signal indicating that the push button 50 has been pressed (hereinafter may be referred to as a "button press signal") is output from the push button 50 to the recognition result extraction unit 13a and the button color control unit 18. When the button press signal is input in response to pressing the push button 50, the recognition result extraction unit 13a extracts the misrecognition result from the recognition result storage unit 12. The push button 50 is an example of an interface (hereinafter may be referred to as an "missorting input interface") that accepts an input indicating that undesired waste has been missorted by the object sorting device 30a. An example of a missorting input interface other than the push button 50 is a software button displayed on a touch panel. That is, the recognition result extraction unit 13a extracts the erroneous recognition result from the recognition result storage unit 12 when an input indicating that undesired waste has been erroneously sorted by the object sorting device 30a is received in the erroneous sorting input interface.

[0021] The input device 70 is operated by an operator, and the operator can use the input device 70 to specify any location in the image of missorted waste present displayed on the display 60. When the arbitrarily specified location in the image of missorted waste present is a display location of label information, the label correction unit 14 enables the operator to correct the label information using the input device 70. The label correction unit 14 outputs the recognition result after the label information in the misrecognition result has been corrected (hereinafter may be referred to as the "corrected recognition result") to the training data storage unit 15.

[0022] Many waste images with feature information added thereto are stored in advance as training data in the training data storage unit 15. The training data storage unit 15 also stores the corrected recognition results output from the label correction unit 14 as new training data.

[0023] The machine learning unit 16 performs machine learning using the training data stored in the training data storage unit 15, and updates the trained model stored in the trained model storage unit 17 with the trained model after machine learning. Therefore, the recognition of waste images by the image recognition unit 11 is performed using the trained model after it has been updated by additional machine learning performed by the machine learning unit 16 using the corrected recognition results.

[0024] On the other hand, the object sorting device 30a outputs to the button color control unit 18 a signal indicating the operating state of the object sorting device 30a (hereinafter, sometimes referred to as a "sorting state signal").

[0025] The button color control unit 18 changes the color of the push button 50 (hereinafter sometimes referred to as "button color") based on the button press signal, the recognition completion signal, the extraction completion signal, and the selection status signal. The push button 50 is configured using, for example, an LED (Light Emitting Diode), and the button color is changeable.

[0026] <Feature information> 3 is a diagram showing an example of feature information according to the first embodiment of the present disclosure. Hereinafter, it is assumed that the waste is empty bottles, and that the object sorting device 30a sorts each empty bottle traveling on the belt conveyor 40 into two types of empty bottles: brown empty bottles (hereinafter sometimes referred to as "brown bottles") and empty bottles of a color other than brown (hereinafter sometimes referred to as "other colors") (hereinafter sometimes referred to as "other-color bottles"). Hereinafter, it is assumed that the desired waste is brown bottles, and training data for brown bottles is stored in the training data storage unit 15.

[0027] As shown in FIG. 3, when a captured image includes an empty bottle image (hereinafter sometimes referred to as an "empty bottle image") BI as a waste image, the image recognition unit 11 recognizes the empty bottle image BI present in the captured image using a learned model stored in the learned model storage unit 17, and assigns feature information including label information LA, contour information CO, and centroid coordinates DG to the recognized empty bottle image BI. If the empty bottle image BI is an image of a brown bottle (hereinafter sometimes referred to as a "brown bottle image"), the image recognition unit 11 assigns type information "brown bottle" to the empty bottle image BI as label information LA. On the other hand, if the empty bottle image BI is an image of a bottle of another color (hereinafter sometimes referred to as an "other color bottle image"), the image recognition unit 11 assigns type information "other color bottle" to the empty bottle image BI as label information LA. Furthermore, the image recognition unit 11 assigns contour information CO to the empty bottle image BI in the captured image. With the long side of the rectangular captured image as the X axis and the short side as the Y axis, the contour information CO is formed by a plurality of coordinate points (x0, y0), (x1, y1), ..., (xn, yn). Furthermore, the image recognition unit 11 calculates the area center of gravity of the empty bottle image BI in the captured image and assigns a center of gravity coordinate DG to the empty bottle image BI. The center of gravity coordinate DG is formed by a single coordinate point (X, Y).

[0028] Here, the multiple coordinate points (x0, y0), (x1, y1), ..., (xn, yn) that form the contour information CO, and the single coordinate point (X, Y) that forms the center of gravity coordinates are coordinate points in the coordinate system of the captured image, i.e., the coordinate system of the camera 20 (hereinafter sometimes referred to as the "camera coordinate system").

[0029] <Tracing the center of gravity> 4, 5, 6, and 7 are diagrams showing examples of traces of the centroid coordinates in the first embodiment of the present disclosure.

[0030] First, as shown in FIG. 4, the image recognition unit 11 recognizes the brown bottle image BO in the captured image acquired at time t11, and calculates the barycentric coordinates DG1 of the brown bottle image BO at time t11.

[0031] 5, the image recognition unit 11 predicts the coordinates EG2 of the center of gravity of the brown bottle image BO in the image captured at time t12, which will be acquired next after the image captured at time t11, based on the frame rate FR [frame / sec] of the camera 20 and the conveying speed CS [mm / sec] of the belt conveyor 40. The coordinates EG2 of the center of gravity at time t12 are predicted to move by CS / FR in the conveying direction CD of the belt conveyor 40 (i.e., the X direction in FIG. 1) per frame relative to the coordinates DG1 of the center of gravity at time t11. The image recognition unit 11 also sets a circular tolerance range TR of a predetermined size centered on the predicted coordinates EG2 of the center of gravity.

[0032] 6, the image recognition unit 11 recognizes the brown bottle image BO in the captured image acquired at time t12, and calculates the barycentric coordinates DG2 of the brown bottle image BO at time t12. The image recognition unit 11 also determines whether the barycentric coordinates DG2 are within the allowable range TR.

[0033] If the center of gravity coordinate DG2 is within the allowable range TR, the image recognition unit 11 determines that the subject of both the brown bin images BO at time t11 and time t12 is the same brown bin, and sets a straight trace line TL connecting the center of gravity coordinate DG1 to the center of gravity coordinate DG2, as shown in Fig. 7. On the other hand, if the center of gravity coordinate DG2 is not within the allowable range TR, the image recognition unit 11 stops tracing the center of gravity coordinates.

[0034] The image recognition unit 11 sequentially executes tracing of the center of gravity coordinates as described above for the captured images sequentially acquired at the frame rate FR.

[0035] <Extraction of desired waste> Fig. 8 is a diagram showing an example of extraction of desired waste in Example 1 of the present disclosure. In Fig. 8, the frame rate FR of the camera 20 is set to a frame rate that allows the same empty bottle being transported in the transport direction CD to be photographed a maximum of three times, based on the transport speed CS of the belt conveyor 40 and the angle of view of the camera 20. In addition, the object sorting device 30a uses a suction pad 31 to extract brown bottles, which are desired waste, from the group of empty bottles.

[0036] In FIG. 8, the image recognition unit 11 recognizes the brown bottle image BO in the captured image acquired at time t11, and calculates the barycentric coordinates DG1 (X1, Y1) of the brown bottle image BO at time t11.

[0037] Next, the image recognition unit 11 recognizes the brown bottle image BO in the captured image acquired at time t12, and calculates the barycentric coordinates DG2 (X2, Y2) of the brown bottle image BO at time t12.

[0038] Next, the image recognition unit 11 determines that the subject of both the brown bin images BO at time t11 and time t12 is the same brown bin, and sets a straight trace line TL connecting the center of gravity coordinates DG1 (X1, Y1) to the center of gravity coordinates DG2 (X2, Y2).

[0039] Next, the image recognition unit 11 recognizes the brown bottle image BO in the captured image acquired at time t13, and calculates the barycentric coordinates DG3 (X3, Y3) of the brown bottle image BO at time t13.

[0040] Next, the image recognition unit 11 determines that the subject of both the brown bin images BO at time t12 and time t13 is the same brown bin, and sets a straight trace line TL connecting the center of gravity coordinates DG1 (X1, Y1) to the center of gravity coordinates DG3 (X3, Y3).

[0041] Next, the image recognition unit 11 calculates the extraction target coordinate TC(X,Y) of the brown bottle by extending the trace line TL connecting the center of gravity coordinate DG1(X1,Y1) to the center of gravity coordinate DG3(X3,Y3) by linear approximation to the installation position of the suction pad 31 in the X direction and converting the camera coordinate system to the coordinate system of the object sorting device 30a. Furthermore, the image recognition unit 11 calculates the time tH (hereinafter sometimes referred to as the "extraction target time") at which the brown bottle, the subject of the brown bottle image BO recognized at times t11, t12, and t13, will arrive at the extraction target coordinate TC(X,Y), based on the conveying speed CS. The image recognition unit 11 transmits a control signal including the calculated extraction target coordinate TC(X,Y) and extraction target time tH to the object sorting device 30a.

[0042] In the object sorting device 30a, the suction pad 31 moves to the extraction target coordinates TC(X,Y) according to the extraction target coordinates TC(X,Y) and extraction target time tH received from the image recognition unit 11, and extracts the empty bottle located at the extraction target coordinates TC(X,Y) from the group of empty bottles at the extraction target time tH. As a result, in the object sorting device 30a, the brown bottle that is the subject of the brown bottle image BO recognized by the image recognition unit 11 at times t11, t12, and t13 is extracted from the group of empty bottles.

[0043] <Control device operation> 9 to 15 are diagrams illustrating an example of the operation of the control device according to the first embodiment of the present disclosure.

[0044] 9, from time ta to time ti, empty bottles BOa, BOb, BOc, BOd, BOe, and BOf are sequentially transported in the transport direction CD by the belt conveyor 40. The empty bottle BOa is a brown bottle, which is desired waste, and the empty bottles BOb, BOc, BOd, BOe, and BOf are other-colored bottles, which are undesired waste.

[0045] As the group of empty bottles is transported, camera 20 captures, at time ta, a captured image CIa that does not include images of empty bottles, at time tb, a captured image CIb that includes brown bottle image Bia, which is an image of brown bottle BOa, at time tc, a captured image CIc that includes brown bottle image Bia and other color bin image BIb, which is an image of other color bottle BOb, at time td, a captured image CId that includes brown bottle image Bia, other color bin image BIb, and other color bin image BIc, which is an image of other color bottle BOc, and at time te, a captured image CIe that includes other color bin image BIb, other color bin image BIc, and other color bin image BId, which is an image of other color bottle BOd. In addition, as the group of empty bottles is transported, camera 20 captures, at time tf, a captured image CIf including other-color bin image BIc, other-color bin image BId, and other-color bin image BIe, which is an image of other-color bin BOe; at time tg, a captured image CIg including other-color bin image BId, other-color bin image BIe, and other-color bin image BIf, which is an image of other-color bin BOf; at time th, a captured image CIh including other-color bin image BIe and other-color bin image BIf; at time ti, a captured image CIi including other-color bin image BIf is captured; and at time tj, a captured image CIj that does not include an image of an empty bottle is captured. Therefore, at time ti, the recognition result memory unit 12 stores each captured image CIa, CIb, CIc, CId, CIe, CIf, CIg, CIh, and CIi with feature information added, in one-to-one correspondence with each captured time ta, tb, tc, td, te, tf, tg, th, and ti.

[0046] Here, it is assumed that at time tb, the image recognition unit 11 correctly recognizes the brown bottle image BIa as a brown bottle image. Then, at time tf, the object sorting device 30a uses the suction pad 31 to extract the brown bottle BOa from the group of empty bottles, and at time tg, moves the suction pad 31 to the release position and releases the brown bottle BOa from the suction pad 31.

[0047] On the other hand, at time td, the image recognition unit 11 erroneously recognizes the other-colored bottle image BIc as a brown bottle image. Therefore, at time th, the object sorting device 30a uses the suction pad 31 to extract the other-colored bottle BOC from the group of empty bottles, and at time ti, moves the suction pad 31 to the release position and releases the other-colored bottle BOC from the suction pad 31. In other words, as a result of the image recognition unit 11 erroneously recognizing the other-colored bottle image BIc as a brown bottle image, the object sorting device 30a erroneously sorts the other-colored bottle BOC as a brown bottle.

[0048] At time ti when the other-colored bin BOC is released from the suction pad 31, the operator notices that the other-colored bin BOC, which is not a brown bin, has been mistakenly selected by the object sorting device 30a and presses the push button 50. Therefore, at time ti, a button press signal is output from the push button 50 to the recognition result extraction unit 13a and the button color control unit 18.

[0049] In response to the button press signal, the recognition result extraction unit 13a determines whether or not there is an erroneous recognition result including a multi-color bin image BIc, which is an image of missorted waste, at time ti. Based on the frame rate FR, the distance XD in the X direction between the camera 20 and the object sorting device 30a, and the conveying speed CS, the recognition result extraction unit 13a determines that, of the captured images CIa, CIb, CIc, CId, CIe, CIf, CIg, CIh, and CIi stored in the recognition result storage unit 12, the captured images CId, CIe, and CIf associated with times td, te, and tf, respectively, are images containing missorted waste. For example, the recognition result extraction unit 13a determines the time XD / CS [sec] before the time ti when the push button 50 is pressed as time te, determines the time ((XD / CS)-(1 / FR)) [sec] before the time ti as time td, and determines the time ((XD / CS)+(1 / FR)) [sec] before the time ti as time tf. The recognition result extraction unit 13a then extracts, as erroneous recognition results, the recognition results including the captured images CId, CIe, and CIf corresponding to the times td, te, and tf from among the captured images CIa, CIb, CIc, CId, CIe, CIf, CIg, CIh, and CIi stored in the recognition result storage unit 12 at time ti. The recognition result extraction unit 13a outputs the captured images CId, CIe, and CIf extracted from the recognition result storage unit 12 together with their feature information to the label correction unit 14 and displays them on the display 60 together with the contours represented by the contour information.

[0050] Because the image recognition unit 11 correctly recognized the brown bin image BIa as a brown bin image at time tb, feature information RRa1 including the correct label information LAa of “brown bin” and correct outline information COa1 is assigned to the brown bin image BIa in the captured image CId, as shown in Fig. 10. Also, because the image recognition unit 11 correctly recognized the other-color bin image BIb as an other-color bin image at time tc, feature information RRb1 including the correct label information LAb of “other-color bin” and correct outline information COb1 is assigned to the other-color bin image BIb in the captured image CId, as shown in Fig. 10. On the other hand, because the image recognition unit 11 erroneously recognized the other-color bin image BIc as a brown bin image at time td, feature information RRc1 including the erroneous label information LAc of “brown bin” and correct outline information COc1 is assigned to the other-color bin image BIc in the captured image CId, as shown in Fig. 10.

[0051] Similarly, as shown in Fig. 11, in the captured image CIe, the other-color bin image BIb is assigned feature information RRb2 including the correct label information LAb of "other-color bin" and the correct outline information COb2, and the other-color bin image BIc is assigned feature information RRc2 including the erroneous label information LAc of "brown bin" and the correct outline information COc2. Furthermore, because the image recognition unit 11 correctly recognized the other-color bin image BId as an other-color bin image at time te, as shown in Fig. 11, the other-color bin image BId in the captured image CIe is assigned feature information RRd1 including the correct label information LAd of "other-color bin" and the correct outline information COd1.

[0052] 12, in the captured image CIf, the other-color bin image BIc is assigned feature information RRc3 including erroneous label information LAc, "brown bin," and correct outline information COc3, and the other-color bin image BId is assigned feature information RRd2 including correct label information LAd, "other-color bin," and correct outline information COd2. Furthermore, because the image recognition unit 11 correctly recognized the other-color bin image BIe as an other-color bin image at time tf, the other-color bin image BIe is assigned feature information RRe1 including correct label information LAe, "other-color bin," and correct outline information COe1, in the captured image CIf, as shown in FIG.

[0053] Fig. 13 shows an example of a display of an erroneous recognition result including an image of missorted waste. For example, a captured image CI as shown in Fig. 13 extracted from the recognition result storage unit 12 by the recognition result extraction unit 13a is displayed on the display 60. The captured image CI includes empty bottle images BI1, BI2, BI3, BI4, and BI5. The empty bottle image BI1 is surrounded by an outline represented by outline information CO1, the empty bottle image BI2 is surrounded by an outline represented by outline information CO2, the empty bottle image BI3 is surrounded by an outline represented by outline information CO3, the empty bottle image BI4 is surrounded by an outline represented by outline information CO4, and the empty bottle image BI5 is surrounded by an outline represented by outline information CO5.

[0054] Here, the correct color of empty bottle images BI1, BI3, and BI4 is brown, and the correct color of empty bottle images BI2 and BI5 is another color. Therefore, the label information LA1, LA3, and LA4, "brown bottle," assigned to empty bottle images BI1, BI3, and BI4, respectively, and the label information LA2, "other-colored bottle," assigned to empty bottle image BI2, are correct, while the label information LA5a, "brown bottle," assigned to empty bottle image BI5 is incorrect. In other words, empty bottle image BI5, which is an image of a other-colored bottle, corresponds to an image of missorted waste, and captured image CI corresponds to an image containing missorted waste, including empty bottle image BI5, which is an image of missorted waste.

[0055] For example, the operator operates the input device 70 to use a pointer PO displayed on the captured image CI to click on the portion of the captured image CI where the label information LA5a, "brown bottle," is displayed on the empty bottle image BI5, as shown in FIG. 14, thereby specifying the label information LA5a for the empty bottle image BI5. When the label information LA5a is specified by operating the input device 70, the label correction unit 14 allows the specified label information LA5a to be corrected. The operator then uses, for example, a keyboard to correct the label information assigned to the empty bottle image BI5 from "brown bottle" to "other-colored bottle," as shown in FIG. 15. Following the operator's correction, the label correction unit 14 corrects the erroneous label information LA5a, "brown bottle," assigned to the empty bottle image BI5, to label information LA5b, "other-colored bottle," indicating the correct type of empty bottle image BI5. In this way, the erroneous label information included in the erroneous recognition result is corrected by the label correction unit 14.

[0056] The minimum required storage capacity of the recognition result storage unit 12 can be calculated according to formula (1). In formula (1), "Ts" is the time [sec] required from when a brown bottle enters the field of view of the camera 20 until the brown bottle that has been sucked onto the suction pad 31 is released from the suction pad 31, "Tj" is the time [sec] required from when an incorrectly selected other-colored bottle is released from the suction pad 31 until the operator presses the push button 50, "FR" is the frame rate of the camera 20, and "DS" is the data size [MB] per captured image. Storage capacity of the recognition result storage unit 12≧(Ts+Tj) / FR×DS (1)

[0057] <Button color transition> 16 and 17 are diagrams illustrating an example of transition of button color in Example 1 of the present disclosure. As shown in Fig. 16 and Fig. 17, the button color transitions between green, red, and yellow, for example.

[0058] 16, when the image recognition unit 11 is in a recognition waiting state from when it outputs the previous recognition completion signal until it outputs the next recognition completion signal, the button color control unit 18 sets the button color to green, which indicates that extraction of an erroneous recognition result is possible. When the button color is green, pressing the push button 50 is valid.

[0059] When the green push button 50 is pressed to input a button press signal, the button color control unit 18 determines that the operator has instructed the start of extraction of erroneous recognition results, and changes the button color from green to yellow. When the button color is yellow, pressing the push button 50 is valid.

[0060] When a button press signal is input by pressing the push button 50 in the yellow state, the button color control unit 18 determines that an instruction to stop extracting erroneous recognition results has been given by the operator, and changes the button color from yellow to green. Alternatively, when the push button 50 is not pressed while the button color is yellow, the button color control unit 18 changes the button color from yellow to green at the point in time when an extraction completion signal is output from the recognition result extraction unit 13a.

[0061] Furthermore, when the button color is green and a recognition completion signal is output from the image recognition unit 11 without the push button 50 being pressed, the button color control unit 18 changes the button color from green to red. When the button color is red, pressing the push button 50 is invalid. After recognition is completed by the image recognition unit 11, the button color control unit 18 maintains the button color red until the suction pad 31 has completely moved to the target position corresponding to the extracted target coordinates. The button color control unit 18 determines the state of the suction pad 31 based on the sorting state signal output from the object sorting device 30a.

[0062] Furthermore, after the suction pad 31 has completed moving to the target position corresponding to the extracted target coordinates, the button color control unit 18 maintains the button color red until the desired waste arrives at the target position. Furthermore, if the button color control unit 18 determines, based on the sorting status signal, that the suction pad 31 has failed to attract the desired waste, it changes the button color from red to green. On the other hand, if the button color control unit 18 determines, based on the sorting status signal, that the suction pad 31 has successfully attracted the desired waste, it maintains the button color red. The button color control unit 18 maintains the button color red while the suction pad 31 is moving to the release position for the desired waste.

[0063] Then, when the button color control unit 18 determines based on the sorting state signal that the suction pad 31 has completed releasing the desired waste at the release position, it changes the button color from red to green.

[0064] FIG. 17 shows the relationship between the transition of button colors and the transition of tasks in the control device 10a.

[0065] In Figure 17, empty bottles BOf and BOg are other-colored bottles that are undesired waste, and empty bottle Boh is a brown bottle that is desired waste. Also, assume that empty bottle BOg is correctly recognized as a other-colored bottle, empty bottle Boh is correctly recognized as a brown bottle, and empty bottle Bof is incorrectly recognized as a brown bottle. Therefore, the control device 10a generates a task TA1 for empty bottle BOf and a task TA2 for empty bottle Boh. A new task is generated when an empty bottle whose center of gravity coordinates have been successfully traced moves out of the field of view of camera 20. Furthermore, an existing task is deleted when the empty bottle is released from suction pad 31 or when suction pad 31 fails to adsorb the empty bottle.

[0066] If no task exists at time t21, the button color control unit 18 sets the button color to green.

[0067] Next, at time t22, when a task TA1 for an empty bottle BOf is generated, the button color control unit 18 sets the button color to green. The button color control unit 18 also maintains the button color to green until time t24 when a task TA2 for an empty bottle BOh is generated.

[0068] Next, at time t24, when a task TA2 for the empty bottle BOh is generated, the button color control unit 18 changes the button color from green to red.

[0069] Next, when task TA1 is deleted at time t25, button color control unit 18 changes the button color from red to green.

[0070] Next, at time t26, a predetermined time after the task TA1 is deleted, the button color control unit 18 changes the button color from green to red.

[0071] Then, when task TA2 is deleted at time t27, button color control unit 18 changes the button color from red to green.

[0072] In addition, if software buttons displayed on a touch panel are used as the error selection input interface instead of push buttons 50, the button color control unit 18 can change the color of the software buttons in the same manner as described above.

[0073] The first embodiment has been described above.

[0074] [Example 2] <Control device configuration> FIG. 18 is a diagram illustrating a configuration example of a control device according to a second embodiment of the present disclosure. A control device 10b illustrated in FIG. 18 corresponds to the control device 10 illustrated in FIG. 1, and an object sorting device 30b illustrated in FIG. 18 corresponds to the object sorting device 30 illustrated in FIG. 1. However, the push button 50 illustrated in FIGS. 1 and 2 is not connected to the control device 10b. Furthermore, the object sorting device 30b differs from the object sorting device 30a illustrated in FIG. 2 in that it does not output a sorting state signal. In FIG. 18, the control device 10b includes an image recognition unit 11, a recognition result storage unit 12, a recognition result extraction unit 13b, a label correction unit 14, a training data storage unit 15, a machine learning unit 16, and a trained model storage unit 17.

[0075] Typically, the size of brown bottles is often smaller than a predetermined size. Furthermore, in the object sorting device 30b, empty bottles corresponding to empty bottle images labeled "brown bottle" are sorted as desired waste. Therefore, the recognition result extraction unit 13b determines that, among the captured images stored in the recognition result storage unit 12, empty bottle images labeled "brown bottle" and having an area equal to or greater than the threshold value TH2 are misclassified waste images. The recognition result extraction unit 13b then extracts, from the recognition result storage unit 12, captured images including empty bottle images labeled "brown bottle" and having an area equal to or greater than the threshold value TH2, along with their feature information, as misrecognition results.

[0076] The second embodiment has been described above.

[0077] [Example 3] <Control device configuration> Fig. 19 is a diagram illustrating a configuration example of a control device according to a third embodiment of the present disclosure. A control device 10c illustrated in Fig. 19 corresponds to the control device 10 illustrated in Fig. 1, and an object sorting device 30c illustrated in Fig. 19 corresponds to the object sorting device 30 illustrated in Fig. 1. However, the push button 50 illustrated in Figs. 1 and 2 is not connected to the control device 10c.

[0078] 19, the control device 10c has an image recognition unit 11, a recognition result storage unit 12, a recognition result extraction unit 13c, a label correction unit 14, a teacher data storage unit 15, a machine learning unit 16, and a learned model storage unit 17. The object sorting device 30c has a suction pad 31. The suction pad 31 has a weight sensor that detects the weight of the empty bottle that is sucked up, and the object sorting device 30c outputs information indicating the weight of the empty bottle sucked up to the suction pad 31 (hereinafter, sometimes referred to as "weight information") to the recognition result extraction unit 13c.

[0079] Typically, the weight of a brown bottle is often less than a predetermined weight. Furthermore, in the object sorting device 30c, empty bottles corresponding to empty bottle images labeled "brown bottle" are sorted as desired waste. Based on the weight information, the recognition result extraction unit 13c determines, among the captured images stored in the recognition result storage unit 12, empty bottle images labeled "brown bottle" that correspond to empty bottles with a weight equal to or greater than the threshold value TH3 as missorted waste images. The recognition result extraction unit 13c then extracts, from the recognition result storage unit 12, captured images including empty bottle images corresponding to empty bottles labeled "brown bottle" and with a weight equal to or greater than the threshold value TH3, along with their feature information, as misrecognition results.

[0080] The third embodiment has been described above.

[0081] [Example 4] The recognition result storage unit 12, the training data storage unit 15, and the trained model storage unit 17 are realized as hardware, for example, by a memory or storage. The image recognition unit 11, the recognition result extraction units 13a, 13b, and 13c, the label correction unit 14, the machine learning unit 16, and the button color control unit 18 are realized as hardware, for example, by a processor such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

[0082] The fourth embodiment has been described above.

[0083] As described above, the object sorting system (object sorting system 1 of the embodiment) of the present disclosure includes a camera (camera 20 of the embodiment), a recognition unit (image recognition unit 11 of the embodiment), a memory unit (recognition result memory unit 12 of the embodiment), a sorting unit (object sorting devices 30, 30a, 30b, and 30c of the embodiment), and an extraction unit (recognition result extraction units 13a, 13b, and 13c of the embodiment). The camera captures a first image, which is an image of a group of objects, on a conveyance path along which the group of objects is conveyed. The recognition unit recognizes a second image, which is an image of each object present in the first image, using a trained model, and assigns feature information, which is information indicating the features of the second image, to the second image, thereby generating a recognition result in which the feature information is assigned to the second image in the first image. The memory unit stores the recognition result. The sorting unit selects a desired object from the group of objects conveyed on the conveyance path based on the recognition result. When the sorting unit erroneously selects an undesired object other than the desired object, the extraction unit determines, from the recognition results stored in the memory unit, an erroneous recognition result that is a recognition result that includes an image of the erroneously selected undesired object, and extracts the erroneous recognition result from the memory unit. For example, the feature information includes information indicating the type of each object in the object group and information indicating the contour of each object in the object group.

[0084] This allows the operator to generate training data by correcting the erroneous recognition results based on the erroneous recognition results extracted from the memory unit, thereby reducing the effort required to generate training data compared to when all annotation work is done manually on image data.

[0085] In addition, the object sorting system of the present disclosure has an interface that accepts input indicating that an undesired object has been erroneously sorted by the sorting unit, and the extraction unit extracts the erroneous recognition result from the memory unit when input is received from the interface.

[0086] Alternatively, the extraction unit extracts the erroneous recognition result from the storage unit based on the area of ​​the second image.

[0087] Alternatively, the extraction unit extracts the erroneous recognition results from the storage unit based on the weight of the desired object selected by the selection unit.

[0088] In this way, erroneous recognition results can be accurately extracted.

[0089] The interface is a button (push button 50 in the embodiment) that accepts input from an operator, and the object sorting system of the present disclosure has a control unit that changes the color of the button depending on the extraction status of erroneous recognition results by the extraction unit.

[0090] This allows the operator to be notified of the extraction status of erroneous recognition results.

[0091] The extraction unit also determines whether the recognition result is incorrect based on the frame rate of the camera, the distance between the camera and the sorting unit, and the transport speed of the object group on the transport path.

[0092] In this way, erroneous recognition results can be accurately determined. [Explanation of symbols]

[0093] 1. Object sorting system 10, 10a, 10b, 10c control device 20 Camera 30, 30a, 30b, 30c Object sorting device 50 push button 11 Image Recognition Unit 12 Recognition result storage unit 13a,13b,13c Recognition result extraction part 14 Label Correction Department 15 Teacher data storage unit 16 Machine Learning Department 17 Trained model memory 18 Button color control section

Claims

1. a camera that captures a first image, which is an image of the object group, on a transport path along which the object group is transported; a recognition unit that recognizes a second image that is an image of each object present in the first image using a trained model, assigns feature information that is information indicating features of the second image to the second image, and generates a recognition result in which the feature information is assigned to the second image in the first image; a storage unit that stores the recognition result; a sorting unit that sorts a first predetermined object from the group of objects conveyed on the conveying path based on the recognition result; an extraction unit that, when a second predetermined object other than the first predetermined object is selected by the selection unit, determines a recognition result that includes an image of the second predetermined object to be selected from the recognition results stored in the storage unit, and extracts the recognition result from the storage unit; Equipped with the extraction unit extracts the recognition result from the storage unit based on an area of ​​the second image. Object sorting system.

2. a camera that captures a first image, which is an image of the object group, on a transport path along which the object group is transported; a recognition unit that recognizes a second image that is an image of each object present in the first image using a trained model, assigns feature information that is information indicating features of the second image to the second image, and generates a recognition result in which the feature information is assigned to the second image in the first image; a storage unit that stores the recognition result; a sorting unit that sorts a first predetermined object from the group of objects conveyed on the conveying path based on the recognition result; an extraction unit that, when a second predetermined object other than the first predetermined object is selected by the selection unit, determines a recognition result that includes an image of the second predetermined object to be selected from the recognition results stored in the storage unit, and extracts the recognition result from the storage unit; Equipped with the extraction unit extracts the recognition result from the storage unit based on a weight of the first predetermined object selected by the selection unit. Object sorting system.

3. an interface that receives an input indicating that the second predetermined object has been sorted by the sorting unit; the extraction unit extracts the recognition result from the storage unit when the input is made to the interface. The object sorting system according to claim 1 or 2.

4. the interface is a button that accepts the input from an operator; The apparatus further includes a control unit that changes the color of the button depending on the extraction status of the recognition result by the extraction unit. The object sorting system according to claim 3 .

5. the extraction unit determines the recognition result based on a frame rate of the camera, a distance between the camera and the sorting unit, and a conveying speed of the object group on the conveying path. The object sorting system according to claim 1 or 2.

6. the feature information includes information indicating the type of each object in the object group and information indicating the contour of each object in the object group; The object sorting system according to claim 1 or 2.

Images