Data acquisition system, data acquisition method, model generation method, and foreign object recovery system

Abstract

To reduce the cost required for recovering foreign substances in waste.SOLUTION: A data collection system according to one aspect of the present invention comprises an imaging apparatus, a detection device, and an information processing apparatus. The information processing apparatus is configured to execute: continuously acquiring, by using the imaging apparatus, picked-up images capturing waste conveyed on a conveyor; detecting, by using the detection device, that an operator manually recovers foreign substances; and when the detection device detects that the operator manually recovers the foreign substance, collecting, of the continuously acquired picked-up images, picked-up images picked up at the time when the foreign substances are present within an imaging range of the imaging apparatus, as training images in machine learning.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a data collection system, a data collection method, a model generation method, and a foreign object recovery system.

Background Art

[0002] Patent Document 1 proposes an inspection system for inspecting articles. Specifically, the inspection system proposed in Patent Document 1 acquires an inspection image of an article and determines whether the article shown in the inspection image is normal or abnormal by an abnormality detection program other than a learned model. Then, the inspection system generates a dataset for machine learning by associating the result of the determination with the inspection image. According to this inspection system, a dataset used for machine learning of a machine learning model can be automatically generated using the result of an abnormality detection program other than a learned model. Therefore, the cost of generating a learned model can be reduced.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The inventors of this case have found that conventional methods proposed in Patent Document 1, etc., have the following problems. Specifically, in the process of processing household waste and other garbage, foreign objects are sometimes collected from the waste to be processed for various reasons. For example, among the waste being transported on a conveyor belt, waste that cannot be processed as is (e.g., recycled, incinerated, landfilled, etc.) may be collected as foreign objects. A typical example is the collection of recyclable waste that cannot be recycled as is (e.g., soiled or full-contents PET bottles, cans, glass bottles, etc.). In another example, as an intermediate process, the collected waste may be compressed for transport. During this intermediate process, waste that is not suitable for compression may be collected as foreign objects. In yet another example, among the waste to be processed such as incineration, landfilling, and intermediate processing, valuable waste (valuable materials) such as waste containing precious metals may be collected as foreign objects.

[0005] The types of waste collected as foreign objects within the processed waste can vary depending on the region, era, purpose, etc. For example, product packaging in household waste may differ from region to region. Also, waste collection methods may differ from region to region (for example, some regions collect paper products as burnable waste, while others collect them as recyclable waste). The waste to be processed may differ from region to region, and within that, the types of waste collected as foreign objects may also differ from region to region. Therefore, it is difficult to comprehensively identify foreign objects in advance, and consequently, it is difficult to create a program to detect such foreign objects. Consequently, currently, manually collecting foreign objects is the most practical solution, but there is the problem that collecting foreign objects from waste is costly.

[0006] In one respect, this invention was made in consideration of these circumstances, and its purpose is to provide a technology that reduces the cost of recovering foreign objects from waste. [Means for solving the problem]

[0007] To solve the above-mentioned problems, the present invention employs the following configuration.

[0008] In other words, a data collection system according to one aspect of the present invention comprises an imaging device positioned to image garbage being transported on a conveyor belt, a detection device for detecting when a worker has manually collected a foreign object from the garbage, and an information processing device. The information processing device is configured to continuously acquire images of the garbage being transported on the conveyor belt using the imaging device, to detect when a worker has manually collected a foreign object using the detection device, and, when it is detected that a worker has manually collected a foreign object, to collect from the continuously acquired images images the images taken at the time when the foreign object is within the imaging range of the imaging device as training images for machine learning.

[0009] According to this configuration, when workers manually collect foreign objects, images of those objects can be automatically collected as training images. In other words, the manual collection of foreign objects can be used as a trigger to automate the collection of training images of foreign objects. Therefore, the cost of collecting training images of waste collected as foreign objects at the site can be reduced. By using the collected training images for machine learning, a trained machine learning model that has acquired the ability to detect those foreign objects can be generated. By using the generated trained machine learning model, at least a part of the work of detecting foreign objects at the site and collecting the detected foreign objects can be automated. Therefore, according to this configuration, the cost of collecting training images to generate a trained machine learning model with such capabilities can be reduced. Thus, the cost of collecting foreign objects from waste can be reduced.

[0010] In the data acquisition system relating to the above aspect, the detection device may be composed of the imaging device, and the detection of the recovery of the foreign object by the detection device may be performed by analyzing the captured image acquired by the imaging device to detect that the foreign object has been recovered. With this configuration, the imaging device can also serve as the detection device, thus reducing the cost of the data acquisition system.

[0011] In the data collection system relating to the above aspect, the imaging device may be positioned so as to include the worker's work area on the conveyor belt within its imaging range. Detecting the recovery of the foreign object may be comprised of: detecting that the worker picked up the object in the work area by analyzing the image captured by the imaging device; continuously monitoring the state in which the worker is holding the object by analyzing the image captured after detecting that the worker has picked up the object; detecting, based on the monitoring results, that the worker moved the object off the conveyor belt while holding it; and detecting that the object has been recovered as the foreign object in response to the worker moving the object off the conveyor belt while holding it. The image captured to be collected as the training image may be an image captured at a time prior to the time the worker picked up the foreign object. With this configuration, the position and time when the worker picked up the foreign object can be obtained. By using this position and time as a reference, good training images of the foreign object that are not overlapped by the worker's hand can be easily collected.

[0012] In the data collection system relating to the above aspect, the foreign object may be collected by placing it into a collection box, and the detection device may consist of a sensor that senses when the foreign object is placed into the collection box. With this configuration, detecting the collection of the foreign object only requires obtaining the sensing result from the sensor. Therefore, This simplifies the information processing required to detect the recovery of foreign objects.

[0013] The embodiments of the present invention are not limited to the data acquisition system described above. One aspect of the present invention may be a model generation method that generates a trained detection model using training images collected by a data acquisition system according to any of the above embodiments. Another aspect of the present invention may be a foreign object recovery system configured to detect foreign objects and recover the detected foreign objects using the trained detection model generated by the model generation method.

[0014] For example, a model generation method according to one aspect of the present invention may be an information processing method in which a computer performs the following actions: acquires a plurality of training images relating to foreign objects in garbage; trains a detection model to acquire the ability to detect the foreign objects in an input image by machine learning using the acquired plurality of training images; and stores the trained detection model generated by the machine learning. At least a portion of the acquired plurality of training images may be training images collected by a data collection system according to any of the above forms. According to this configuration, by using the collected training images for machine learning, a trained machine learning model that has acquired the ability to detect the foreign objects can be generated.

[0015] Furthermore, for example, a foreign object recovery system according to one aspect of the present invention may include an imaging device, a recovery device, and a control device. The imaging device may be positioned to image garbage being transported on a conveyor belt. The control device may be configured to acquire a target image of the garbage being transported on the conveyor belt using the imaging device, to detect whether or not a foreign object is present in the acquired target image using the trained detection model generated by the model generation method, and, if the detection results in the presence of a foreign object in the acquired target image, to drive the recovery device to recover the foreign object. With this configuration, at least part of the work of detecting foreign objects at the site and recovering the detected foreign objects can be automated by using the generated trained machine learning model.

[0016] Furthermore, as an alternative form of the data collection system relating to each of the above forms, it may be an information processing device, an information processing method, a program, or a computer or other device, machine, etc., that stores such a program. As an alternative form of the model generation method relating to each of the above forms, it may be an information processing system, an information processing device, a program, or a storage medium that stores such a program, that implements at least a part of each of the above configurations. Also, as an alternative form of the foreign object recovery system relating to each of the above forms, it may be an information processing device, an information processing method, a program, or a storage medium that stores such a program, that implements at least a part of each of the above configurations. Here, a computer or the like, readable storage medium is a medium that stores information such as programs by electrical, magnetic, optical, mechanical, or chemical action.

[0017] For example, a data collection method according to one aspect of the present invention may be an information processing method executed by a data collection system including an imaging device arranged to image garbage conveyed by a conveyor, a detection device for detecting that an operator has manually collected foreign matter among the garbage, and an information processing device. The data collection method may include the information processing device continuously acquiring an imaging image of the garbage reflected on the conveyor by the imaging device, detecting by the detection device that the operator has manually collected foreign matter, and when it is detected that the operator has manually collected the foreign matter, collecting, as a training image in machine learning, an imaging image captured at a time when the foreign matter is present within the imaging range of the imaging device among the continuously acquired imaging images.

Advantages of the Invention

[0018] According to the present invention, it is possible to reduce the cost of collecting foreign matter in garbage.

Brief Description of the Drawings

[0019] [Figure 1] FIG. 1 schematically shows an example of a scene to which the present invention is applied. [Figure 2] FIG. 2 schematically shows an example of the hardware configuration of an information processing device according to an embodiment. [Figure 3] FIG. 3 schematically shows an example of the hardware configuration of a model generation device according to an embodiment. [Figure 4] FIG. 4 schematically shows an example of the hardware configuration of a control device according to an embodiment. [Figure 5] FIG. 5 schematically shows an example of the software configuration of an information processing device according to an embodiment. 8] [Figure 6] FIG. 6 schematically shows an example of the software configuration of a model generation device according to an embodiment. [Figure 7] FIG. 7 schematically shows an example of the software configuration of a control device according to an embodiment. [Figure 8]FIG. 8 is a flowchart showing an example of the operation procedure of the entire system according to the embodiment. [Figure 9] FIG. 9 is a flowchart showing an example of the processing procedure of the data collection system (information processing apparatus) according to the embodiment. [Figure 10] FIG. 10 is a flowchart showing an example of a subroutine of the process for detecting foreign object collection according to the embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of the processing procedure of the model generation apparatus according to the embodiment. [Figure 12] FIG. 12 is a flowchart showing an example of the processing procedure of the foreign object collection system (control apparatus) according to the embodiment. [Figure 13] FIG. 13 schematically shows an example of another scenario to which the present invention is applied.

Embodiments for Carrying Out the Invention

[0020] Hereinafter, an embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described based on the drawings. However, the present embodiment described below is merely an exemplification of the present invention in every aspect. Various improvements or modifications may be made without departing from the scope of the present invention. In carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted. Note that the data appearing in the present embodiment is described in natural language, but more specifically, it is specified by a quasi-language, command, parameter, machine language, etc. that can be recognized by a computer.

[0021] §1 Application Example FIG. 1 schematically shows an example of a scenario to which the present invention is applied. The system shown in FIG. 1 includes a data collection system 100, a model generation apparatus 2, and a foreign object collection system 300.

[0022] [Data Collection System] The data acquisition system 100 is configured to collect training images that can be used for machine learning to acquire the ability to detect foreign objects in garbage. In this embodiment, the data acquisition system 100 comprises an imaging device C1, a detection device, and an information processing device 1. The imaging device C1 is positioned to image garbage A being transported on a conveyor V. The detection device is The system is configured to detect when a worker P manually collects a foreign object B from the waste A being transported by conveyor V.

[0023] The information processing device 1 is connected to the imaging device C1 and the detection device. The method of connecting each device is not particularly limited and may be appropriately selected depending on the embodiment. The information processing device 1 is one or more computers configured to perform information processing related to the acquisition of training images. In this embodiment, the information processing device 1 is configured to perform the following information processing.

[0024] Specifically, the information processing device 1 continuously acquires captured images 121 of waste A being transported on the conveyor belt V by the imaging device C1. The information processing device 1 detects that worker P has manually collected foreign object B using a detection device. When it is detected that worker P has manually collected foreign object B, the information processing device 1 collects the captured images 121 that were taken at the time when foreign object B was within the imaging range of the imaging device C1, from among the continuously acquired captured images 121, as training images 125 for machine learning.

[0025] The configuration of the detection device is not particularly limited and can be determined as appropriate depending on the embodiment, as long as it is possible to detect that worker P has recovered foreign object B. In this embodiment, the detection device is composed of an imaging device C1. The detection of the recovery of foreign object B by the detection device is performed by analyzing the captured image 121 acquired by the imaging device C1 to detect that foreign object B has been recovered. As a result, in this embodiment, the imaging device C1 can also serve as the detection device, thereby reducing the cost of the data acquisition system 100.

[0026] [Model Generator] The model generation device 2 is one or more computers configured to perform information processing related to the generation of a trained model (detection model 5) through machine learning. In this embodiment, the model generation device 2 is configured to perform the following information processing.

[0027] Specifically, the model generation device 2 acquires multiple training images 221 relating to foreign object B in the waste A. At least some of the acquired training images 221 may be training images 125 collected by the data collection system 100 (information processing device 1). The model generation device 2 trains the detection model 5 using machine learning with the acquired training images 221 to acquire the ability to detect foreign object B in the input image. The model generation device 2 stores the trained detection model 5 generated by machine learning.

[0028] [Foreign object recovery system] The foreign object recovery system 300 is configured to use a trained detection model 5 to detect foreign objects B among the waste A being transported on the conveyor V and to recover the detected foreign objects B. In this embodiment, the foreign object recovery system 300 comprises an imaging device C2, a recovery device R, and a control device 3. The imaging device C2 is positioned to image the waste A being transported on the conveyor V. The recovery device R is configured to recover objects (foreign objects B) being transported on the conveyor V.

[0029] The control device 3 is connected to the imaging device C2 and the retrieval device R. The method of connecting each device is not particularly limited and may be appropriately selected depending on the embodiment. The control device 3 is one or more computers configured to perform information processing to control the operation of the retrieval device R. In this embodiment, the control device 3 is configured to perform the following information processing.

[0030] In other words, the control device 3 captures images of the waste A being transported on the conveyor V by the imaging device C2. The target image 321 is acquired. The control device 3 uses the trained detection model 5 generated by the model generation device 2 to detect whether or not foreign object B is present in the acquired target image 321 (i.e., whether or not foreign object B is visible). If the detection results in the presence of foreign object B in the acquired target image 321, the control device 3 drives the retrieval device R to retrieve the foreign object B.

[0031] [Features] In this embodiment, the data collection system 100 can automatically collect images of foreign objects B as training images 125 in response to worker P manually collecting the foreign object B. In other words, the manual collection of foreign object B can be used as a trigger to automate the collection of training images 125 of foreign object B. Therefore, the cost of collecting training images of waste collected as foreign objects at the site can be reduced. In the model generation device 2, at least a portion of the collected training images 125 can be used for machine learning to generate a trained machine learning model (detection model 5) that has acquired the ability to detect the foreign object. In the foreign object collection system 300, at least a portion of the work of detecting foreign objects at the site and collecting the detected foreign objects can be automated by using the generated trained machine learning model (detection model 5). Therefore, according to the data collection system 100 of this embodiment, the cost of collecting training images for generating a trained machine learning model with such ability can be reduced. As a result, the system of this embodiment can reduce the cost of collecting foreign objects in waste.

[0032] (garbage / foreign object) The type of waste A is not particularly limited, as long as it is something that could be subject to foreign matter contamination, and may be determined appropriately depending on the embodiment. In a typical example, waste A may be household waste such as recyclable waste.

[0033] Foreign matter B may be waste A that is transported on the conveyor V (line) that cannot be processed as is, or waste that should not be processed as is. Processing may include, for example, recycling, intermediate processing, landfill, or incineration. Waste that cannot be processed as is may be, for example, soiled or full-contents plastic bottles, cans, glass bottles, etc. Waste that should not be processed as is may be, for example, valuable waste (valuable material) that can be reused as is.

[0034] In a typical example, at least the data collection system 100 and the foreign object recovery system 300 of the system according to this embodiment may be operated in a recycling facility, an intermediate processing facility, or a final processing facility. Waste A may be an object that is transported by a conveyor and can be subjected to some kind of processing (e.g., recycling, intermediate processing (compression, etc.), landfill, incineration, etc.). Foreign object B may be an object that is not subject to processing (unsuitable object, valuable material, etc.). Waste A and foreign object B may vary depending on the region, era, purpose, etc.

[0035] (Conveyor) The type and configuration of the conveyor V are not particularly limited, as long as it can transport waste A (foreign matter B), and may be appropriately selected depending on the embodiment. In one example, the conveyor V may be a belt conveyor. The path of the conveyor V may be determined arbitrarily. The path of the conveyor V may be configured to be at least partially a loop.

[0036] In one example, the conveyor V may have a control device separate from the system. The control device may be one or more computers configured to control the operation of the conveyor V. In another example, either the information processing device 1 or the control device 3 may also serve as the control device for the conveyor V. The operation of the conveyor V may be controlled separately from the system, or This may be controlled in conjunction with the above system.

[0037] As shown in Figure 1, it is preferable that the foreign object recovery system 300 is located on the same conveyor V (line) as the data collection system 100, and further upstream of the conveyor V than the data collection system 100. This reduces the probability of foreign objects being mixed into the waste, as even if the foreign object recovery system 300 overlooks a foreign object, a manual foreign object recovery step is available downstream. In addition, the data collection system 100 can collect training images 125 of foreign objects B that cannot be automatically recovered by the foreign object recovery system 300. By using the collected training images 125 in the machine learning (which may be retraining / additional training) of the model generation device 2, a trained detection model 5 capable of detecting the foreign object B can be generated. By installing the generated trained detection model 5 in the foreign object recovery system 300, it becomes possible to automatically recover foreign objects B that were previously unrecoverable. In other words, by repeating this series of cycles, the system according to this embodiment can acquire the ability to automatically recover new foreign objects B while operating the foreign object recovery operation. This reduces the number of workers P required to retrieve foreign object B, and as a result, lowers the cost associated with retrieving foreign object B.

[0038] (Imaging device) The type of imaging device (C1, C2) is not particularly limited, as long as it is possible to acquire images of dust A and foreign object B, and may be appropriately selected depending on the embodiment. The imaging devices (C1, C2) may be, for example, a general RGB camera, a stereo camera, an infrared camera, etc.

[0039] Furthermore, the arrangement and orientation of the imaging devices (C1, C2) are not particularly limited, as long as they can image the waste A being transported on the conveyor V, and can be appropriately determined according to the embodiment. In one example, the imaging devices (C1, C2) may be arranged to image the waste A from directly above the conveyor V. This reduces the possibility of waste A overlapping in the obtained images (image 121, target image 321), making it easier to obtain images in which foreign objects B are clearly visible. As a result, by using the obtained training images 125 for machine learning, it can be expected that a trained detection model 5 with high accuracy in detecting foreign objects B will be generated. In addition, the foreign object collection system 300 can be expected to improve the accuracy of automatic collection of known foreign objects (i.e., learned foreign objects). In another example, in order to avoid the worker P's hand overlapping with the waste A as much as possible, the imaging device C2 may be arranged to image the waste A from the opposite direction (for example, diagonally) from the direction in which the worker P extends their hand onto the conveyor V.

[0040] Furthermore, in order to demonstrate the detection capability of the trained detection model 5, it is desirable that there be little discrepancy between the learning environment (i.e., the environment in which training images are obtained) and the inference environment (i.e., the environment in which the trained model is used). In this embodiment, the learning environment is the environment in which the model generation device 2 obtains the training images 221 to be used, and the data acquisition system 100 collects the training images 125. The inference environment is the environment in which the foreign object recovery system 300 detects the foreign object B with the trained detection model 5. For this reason, it is preferable that the imaging conditions of the imaging device C2 are the same as or similar to the imaging conditions of the imaging device C1. The imaging conditions may include, for example, ambient lighting conditions, distance to the conveyor V, orientation, etc. Physical imaging conditions such as brightness, distance, and orientation may be corrected by image processing in at least one of the training images 221 and the target image 321.

[0041] (Collection method) The method by which worker P collects foreign object B is not particularly limited and may be determined as appropriate depending on the embodiment. In a typical example, as shown in Figure 1, a collection box RB may be provided in worker P's work area as a collection point for foreign object B. The configuration of the collection box RB (e.g., type, structure, shape, etc.) is not particularly limited as long as it can collect foreign object B. This may be determined as appropriate depending on the embodiment. The collection box RB may be configured in the shape of a bag or in the shape of a box.

[0042] (Image analysis to detect recovery) In this embodiment, the imaging device C1 also serves as a detection device. Therefore, the imaging device C1 is positioned to capture at least a portion of the worker P's retrieval actions. In one example, the imaging range of the imaging device C1 may include, in addition to the area on the conveyor V (the area where waste A is transported), at least a portion from the worker P's work area on the conveyor V to the retrieval destination (e.g., the retrieval box RB). By analyzing the worker P's actions captured in this portion using image processing, it is possible to estimate whether or not the worker P has retrieved the foreign object B.

[0043] If it is possible to detect that worker P has recovered foreign object B, the specific processing content of the image analysis may be appropriately determined according to the embodiment. The image analysis for detecting that foreign object B has been recovered may consist of general image processing such as pattern matching, or image processing using a trained machine learning model. Simply put, the recovery of foreign object B may be detected by image analysis when worker P's hand holds the object. Alternatively, the recovery of foreign object B may be detected by image analysis when worker P holds the object and carries the object to the recovery destination without letting go of it on the conveyor V (i.e., without letting it flow on the conveyor again).

[0044] In one example, the imaging device C1 may be positioned so that its imaging range includes the work area of ​​worker P on the conveyor V. The work area is the area in which worker P can perform the task of retrieving foreign object B. The work area may be definitively defined or it may be defined approximately. The work area may be a hypothetical area defined based on the assumption that worker P can take action to retrieve foreign object B. The imaging range of the imaging device C1 including the work area may be achieved by capturing at least a part of the worker P's action of retrieving the object being transported on the conveyor V in the captured image 121.

[0045] Detecting the recovery of foreign object B through image analysis may be comprised of the following information processing. Specifically, the information processing device 1 may detect that worker P has picked up an object in the work area by performing image analysis on the captured image 121 acquired by the imaging device C1. After detecting that worker P has picked up an object, the information processing device 1 may continuously monitor the state in which worker P has picked up the object by performing image analysis on the acquired captured image 121. Based on the monitoring results, the information processing device 1 may detect that worker P has moved the object off the conveyor belt V while holding it. Then, in response to worker P moving the object off the conveyor belt V while holding it, the information processing device 1 may detect that the object has been recovered as foreign object B.

[0046] The movement of a hand off the conveyor belt V may be detected at any point in time when worker P picks up an object on the conveyor belt V and carries it to its destination. In one example, the movement of a hand off the conveyor belt V may mean that the hand moves to a position away from the space above the conveyor belt V (i.e., the hand is no longer in the air above the conveyor belt V). The movement of a hand off the conveyor belt V may be detected when the hand has completely left the air above the conveyor belt V. In another example, the movement of a hand off the conveyor belt V may be detected when the hand is still on the conveyor belt V but is in a state where it is highly likely to leave the conveyor belt V. A state where it is highly likely to leave may be, for example, when the hand has moved to the edge of the conveyor belt V and is moving toward the outside of the conveyor belt V.

[0047] The detection method may be set as appropriate depending on the embodiment. In one example, when the hand moves off the conveyor V, it is detected when the worker P's hand moves out of a predetermined range in the captured image 121. It may be detected in this way. In another example, the imaging device C1 may be positioned to image the waste A from directly above the conveyor V, and the imaging range may match or approximate the width of the conveyor V. In this case, the movement of the hand off the conveyor V may be detected when the hand moves out of frame while holding an object.

[0048] If the recovery of foreign object B is detected by the above information processing, the image collected as training image 125 may be an image taken at a time prior to the time when worker P picked up the object that is foreign object B. The image collected as training image 125 is image 121 taken at a time when foreign object B would likely be present within the imaging range of imaging device C1. The earlier time may be a time when foreign object B would likely be present within the imaging range of imaging device C1, and when worker P's hand would not be overlapping foreign object B in image 121.

[0049] The information processing device 1 may use the initial position and time at which it detects that worker P's hand has touched the foreign object B (picked up the object) as a reference, and obtain an image of the foreign object B near that reference position by going back a little from that reference time. In one example, the time to go back may be a predetermined value. In another example, the time to go back may be calculated according to the speed of the conveyor V, the speed of the hand's movement, etc. The speed of the conveyor V may be given as a set value. The speed of the hand's movement may be measured by detecting the hand through image analysis in the continuously acquired captured images 121 and monitoring the state of the hand. In addition, the speed of the hand's movement may be detected by any sensor (e.g., an acceleration sensor).

[0050] According to the detection method using the image analysis described above, the position and time when worker P first picked up foreign object B can be used as a reference to determine which images to collect as training images 125. This makes it easy to collect images as training images 125 in which worker P's hand does not overlap and foreign object B is clearly visible. By using the collected training images 125 for machine learning of the detection model 5, it is possible to generate a trained detection model 5 that can accurately detect foreign object B.

[0051] (Detection model) The detection model 5 (machine learning model) has one or more computational parameters for performing the computational processing required to solve the inference task. The values ​​of the computational parameters of the detection model 5 are adjusted (optimized) through machine learning, thereby enabling the trained detection model 5 to acquire the ability to adequately perform the inference task within the range of the training data used for machine learning.

[0052] In this embodiment, the inference task is to detect foreign objects. Simply put, the inference task may be to determine the presence or absence of a foreign object. In one example, the inference task may include determining the type of foreign object. The inference task may also include detecting the location (e.g., range of presence) of the foreign object within the image. The location of the foreign object may be represented, for example, by a bounding box.

[0053] The type, configuration, and structure of the detection model 5 (machine learning model) are not particularly limited and may be appropriately selected depending on the embodiment. The detection model 5 may be composed of, for example, a neural network, a support vector machine, a regression model, a decision tree model, etc. Any form of machine learning may be adopted, such as supervised learning, unsupervised learning, or reinforcement learning. In any case, training the detection model 5 may be composed of adjusting (optimizing) the values ​​of the computational parameters included in the detection model 5 so that an output (inference result) that fits the training image 221 is derived from the training image 221. The method of adjusting computational parameters in machine learning may be appropriately selected depending on the type and configuration of the machine learning model adopted. For example, methods such as backpropagation and solving optimization problems may be adopted as adjustment methods.

[0054] In a typical example, the detection model 5 may be composed of a neural network, and supervised learning may be employed for machine learning. In this case, each training image 221 is assigned a correct label. That is, the model generator 2 acquires multiple datasets, each consisting of a combination of training images 221 and correct labels. The training images 125 may be assigned correct labels at any time between their collection and use in machine learning. In a typical example, the correct labels may be assigned to the training images 125 when they are collected.

[0055] The ground truth label is configured to indicate the correct (true) answer to the inference task for the training image 221. The content of the ground truth label may be determined as appropriate depending on the capabilities to be acquired by the detection model 5. Simply put, the ground truth label may be configured to indicate the presence or absence of a foreign object. If the detection model 5 is to acquire the ability to identify foreign objects, the ground truth label may be configured to indicate the category of the foreign object. The category of foreign object may include a category indicating the absence of a foreign object. If the detection model 5 is to acquire the ability to detect the range of foreign object existence, the ground truth label may be configured to indicate the range in which the foreign object exists.

[0056] The method for assigning correct labels may be determined as appropriate depending on the embodiment. In one example, correct labels may be assigned automatically, at least partially. For example, the information processing device 1 may automatically assign a correct label indicating the presence of a foreign object to the training image 125, depending on that it was collected as a training image 125. In another example, correct labels may be assigned manually.

[0057] As an example of the adjustment method, the model generator 2 inputs the training images 221 of each dataset into the detection model 5 and performs forward propagation calculations of the detection model 5. As a result of this calculation, the model generator 2 obtains output values ​​from the detection model 5 that correspond to the inference results (foreign object detection results) for the training images 221. The model generator 2 calculates the error between the obtained output values ​​and the true values ​​indicated by the corresponding ground truth labels, and further calculates the gradient of the calculated error. The model generator 2 calculates the error in the values ​​of the calculation parameters of the detection model 5 by backpropagating the calculated error gradient using the backpropagation method. The model generator 2 updates the values ​​of the calculation parameters based on the calculated error.

[0058] Through this series of update processes, the model generator 2 adjusts the values ​​of the calculation parameters of the detection model 5 for each dataset so that the sum of errors between the output value (inference result) obtained by providing the training image 221 and the true value indicated by the correct label is minimized. This adjustment of the calculation parameter values ​​may be repeated until predetermined conditions are met, such as adjusting the set number of iterations or the calculated sum of errors falling below a threshold. Furthermore, machine learning conditions such as the objective function (cost function, loss function, error function) for calculating the error, the learning rate, and the optimization algorithm may be set as appropriate depending on the embodiment. Through this machine learning process, the model generator 2 can train the detection model 5 so that the result of performing the inference task with the training image 221 by the detection model 5 fits the true value indicated by the corresponding correct label for each dataset. As a result, a trained detection model 5 that has acquired the ability to detect foreign objects can be generated.

[0059] In another example, unsupervised learning may be used as the form of machine learning. In a typical example, each training image 221 may consist of images showing foreign objects. The model generation device 2 may use machine learning to generate a one-class classifier as a trained detection model 5 using multiple training images 221 showing foreign objects. The one-class classifier may consist of, for example, a neural network, a support vector machine, etc. The method for generating the trained one-class classifier may be appropriately selected depending on the embodiment. In one example, machine learning may be used A class classifier can acquire the ability to output non-outlier values ​​for images containing foreign objects, while outputting outliers for images without foreign objects. Therefore, a trained one-class classifier can detect foreign objects depending on whether the output of the trained one-class classifier for an input image is an outlier or not. Any method, including known methods, may be used to determine whether an outlier is present.

[0060] In another example, the model generation device 2 may train an autoencoder using training images 221 containing foreign objects. The autoencoder consists of an encoder and a decoder. The encoder and decoder may consist of, for example, a neural network. In one example, training the autoencoder in machine learning is performed by using each training image 221 as training data (input data) and correct labels (teacher signals). The method for adjusting the computational parameters of the autoencoder may be the same as the method for supervised learning described above. According to this machine learning, similar to the one-class discriminator described above, the encoder can acquire the ability to output non-outlier values ​​for images containing foreign objects, while outputting outliers for images without foreign objects. Therefore, a trained encoder can be obtained as a trained detection model 5.

[0061] As another example, the model generation device 2 may calculate principal component vectors by performing principal component analysis using the training image 221 containing foreign objects. Similar to the one-class classifier described above, the calculated principal component vectors can be used to map images containing foreign objects to non-outlier values ​​in the subspace, while images without foreign objects can be mapped to outliers in the subspace. Therefore, the model generation device 2 may construct a one-class classifier using the calculated principal component vectors and obtain the constructed one-class classifier as a trained detection model 5.

[0062] When reinforcement learning is adopted as the form of machine learning, the values ​​of the computational parameters of the detection model 5 may be adjusted using the training image 221 to acquire the ability to detect foreign objects, similar to the forms described above. The conditions for reinforcement learning are not particularly limited as long as the ability to detect foreign objects can be acquired, and may be determined as appropriate depending on the embodiment.

[0063] When detection model 5 is constructed using a neural network, examples of computational parameters include the weights of the connections between each node (neuron) and the threshold values ​​of each node. The structure of the neural network may be defined, for example, by the number of layers, the type of each layer, the configuration of each layer, the number of nodes in each layer, and the connection relationships of the nodes. The structure of the neural network is not particularly limited and may be determined as appropriate depending on the embodiment. Detection model 5 may be constructed using, for example, a fully connected neural network, a convolutional neural network, a recurrent neural network, or a combination thereof.

[0064] (Recovery device) The recovery device R is configured to recover objects (foreign matter B) being transported on the conveyor V. The configuration of the recovery device R is not particularly limited and may be determined appropriately depending on the embodiment, as long as it can selectively recover foreign matter B from waste A.

[0065] In one example, the retrieval device R may be a robotic device (for example, an industrial robot such as an articulated robot). The control device 3 may provide the robotic device (retrieval device R) with positional information of the foreign object B that is visible in the target image 321, thereby causing the robotic device to retrieve the foreign object B. The robotic device may be driven by any method (for example, a known method).

[0066] In another example, the recovery device R may consist of a recovery mechanism such as a trap (e.g., a rake, suction, etc.). The control device 3 detects when a foreign object B visible in the target image 321 passes through the recovery mechanism. The foreign object B may be collected by the collection mechanism by driving it at the right moment. The timing at which the foreign object B passes through the collection mechanism may be appropriately determined depending on the position of the foreign object B in the target image 321, the arrangement of the collection mechanism relative to the imaging device C2, etc.

[0067] (System Configuration) In one example, as shown in Figure 1, the information processing device 1 (data acquisition system 100), the model generation device 2, and the control device 3 (foreign object recovery system 300) may be connected to each other via a network. The type of network may be appropriately selected from, for example, the Internet, wireless communication network, mobile communication network, telephone network, dedicated network, etc. However, the method of exchanging data between each device is not limited to this example and may be appropriately selected depending on the embodiment. In another example, data may be exchanged using a storage medium.

[0068] Furthermore, in the example shown in Figure 1, the information processing device 1, the model generation device 2, and the control device 3 are each separate computers. However, the configuration of the system according to this embodiment is not limited to this example and may be determined as appropriate depending on the embodiment. In another example, at least one combination of the information processing device 1, the model generation device 2, and the control device 3 may be configured by a single computer. In yet another example, at least one of the information processing device 1, the model generation device 2, and the control device 3 may be configured by multiple computers.

[0069] §2 Example Configuration [Hardware configuration] <Information Processing Device> Figure 2 schematically shows an example of the hardware configuration of the information processing device 1 according to this embodiment. The information processing device 1 according to this embodiment is a computer in which a control unit 11, a storage unit 12, a communication interface 13, an external interface 14, an input device 15, an output device 16, and a drive 17 are electrically connected.

[0070] The control unit 11 is a hardware processor, a CPU (Central Processing Unit), It includes RAM (Random Access Memory), ROM (Read Only Memory), etc., and is configured to perform information processing based on programs and various data. The control unit 11 (CPU) is an example of processor resources.

[0071] The memory unit 12 is an example of a memory resource. The memory unit 12 is also an example of a storage medium for storing programs. The memory unit 12 may be composed of, for example, a hard disk drive, a solid-state drive, etc. In this embodiment, the memory unit 12 stores various information such as the data acquisition program 81 and training images 125.

[0072] The data acquisition program 81 is a program that causes the information processing device 1 to perform information processing (Figures 9 and 10, described later) related to the acquisition of training images 125 that can be used for machine learning to acquire the ability to detect foreign objects B. The data acquisition program 81 includes a series of instructions for said information processing. In this embodiment, the training images 125 are acquired as a result of executing the data acquisition program 81.

[0073] The communication interface 13 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, etc., and is an interface for wired or wireless communication over a network. The information processing device 1 may communicate data with other computers (for example, the model generation device 2) via the communication interface 13.

[0074] External interface 14 may include, for example, a USB (Universal Serial Bus) port, a dedicated port These are interfaces for connecting to external devices. The type and number of external interfaces 14 can be arbitrarily selected. In this embodiment, the information processing device 1 may be connected to the imaging device C1 and the detection device via at least one of the communication interface 13 and the external interface 14. The information processing device 1 may be directly connected to at least one of the imaging device C1 and the detection device, or it may be connected indirectly via another computer. If the operation of the conveyor V is to be controllable, the information processing device 1 may be connected to the conveyor V via at least one of the communication interface 13 and the external interface 14.

[0075] The input device 15 is, for example, a device for inputting data such as a mouse or keyboard. The output device 16 is, for example, a device for outputting data such as a display or speaker. The operator can operate the information processing device 1 by using the input device 15 and the output device 16. The input device 15 and the output device 16 may be integrated into a single unit, such as a touch panel display.

[0076] Drive 17 is a drive device for reading various information such as programs stored in the storage medium 91. The data acquisition program 81 may be stored in the storage medium 91. The storage medium 91 is a medium that stores information such as programs by electrical, magnetic, optical, mechanical, or chemical means so that computers and other devices, machines, etc., can read the stored information such as programs. The information processing device 1 may acquire the data acquisition program 81 from the storage medium 91. The training images 125 may be stored in the storage medium 91.

[0077] In Figure 2, a disk-type storage medium such as a CD or DVD is shown as an example of a storage medium 91. However, the type of storage medium 91 is not limited to disk type and may be other types. Examples of storage media other than disk type include semiconductor memory such as flash memory. The type of drive 17 may be appropriately selected according to the type of storage medium 91.

[0078] Regarding the specific hardware configuration of the information processing device 1, components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. Hardware processors include microprocessors, FPGAs (field-programmable gate arrays), DSPs (digital signal processors), and GPs. It may consist of a U (Graphics Processing Unit), etc. The storage unit 12 may consist of RAM and ROM included in the control unit 11. At least one of the communication interface 13, external interface 14, input device 15, output device 16, and drive 17 may be omitted. The information processing device 1 may consist of multiple computers. In this case, the hardware configuration of each computer may or may not be the same. The information processing device 1 may consist of a computer designed specifically for the services to be provided, as well as a general-purpose server device, a general-purpose PC (Personal Computer), an industrial PC, a terminal device (for example) For example, a tablet PC or similar device may be used.

[0079] <Model Generator> Figure 3 schematically shows an example of the hardware configuration of the model generation device 2 according to this embodiment. The model generation device 2 according to this embodiment is a computer in which a control unit 21, a storage unit 22, a communication interface 23, an external interface 24, an input device 25, an output device 26, and a drive 27 are electrically connected.

[0080] The control units 21 to drive 27 and the storage medium 92 of the model generation device 2 may be configured in the same way as the control units 11 to drive 17 and the storage medium 91 of the information processing device 1. The control unit 21 includes a hardware processor such as a CPU, RAM, ROM, etc. It is configured to perform various information processing based on the program and data. The storage unit 22 may be composed of, for example, a hard disk drive, a solid-state drive, etc. In this embodiment, the storage unit 22 stores various information such as the model generation program 82, a plurality of training images 221, and learning result data 225.

[0081] The model generation program 82 is a program that causes the model generation device 2 to perform information processing (Figure 11, described later) related to the generation of a trained detection model 5 by machine learning. The model generation program 82 includes a series of instructions for said information processing. The learning result data 225 is configured to show information related to the generated trained detection model 5. In this embodiment, the learning result data 225 is generated as a result of executing the model generation program 82. At least one of the model generation program 82 and the plurality of training images 221 may be stored in the storage medium 92. The model generation device 2 may retrieve at least one of the model generation program 82 and the plurality of training images 221 from the storage medium 92. The learning result data 225 may be stored in the storage medium 92.

[0082] The model generation device 2 may communicate data with other computers (e.g., information processing device 1, control device 3) via the communication interface 23. The model generation device 2 may be connected to an imaging device for acquiring training images 221 via at least one of the communication interface 23 and the external interface 24. The model generation device 2 may accept operations and inputs from an operator using the input device 25 and the output device 26.

[0083] Regarding the specific hardware configuration of the model generation device 2, components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control unit 21 may include multiple hardware processors. The hardware processors may consist of microprocessors, FPGAs, DSPs, GPUs, etc. The storage unit 22 may consist of RAM and ROM included in the control unit 21. At least one of the communication interface 23, external interface 24, input device 25, output device 26, and drive 27 may be omitted. The model generation device 2 may consist of multiple computers. In this case, the hardware configuration of each computer may or may not be the same. Furthermore, the model generation device 2 may be an information processing device designed specifically for the services provided, as well as a general-purpose server device, a general-purpose PC, an industrial PC, a terminal device, etc.

[0084] <Control device> Figure 4 schematically shows an example of the hardware configuration of the control device 3 according to this embodiment. The control device 3 according to this embodiment is a computer in which a control unit 31, a storage unit 32, a communication interface 33, an external interface 34, an input device 35, an output device 36, and a drive 37 are electrically connected.

[0085] The control unit 31 to the drive 37 and the storage medium 93 of the control device 3 may be configured in the same way as the control unit 11 to the drive 17 and the storage medium 91 of the information processing device 1. The control unit 31 includes a hardware processor such as a CPU, RAM, and ROM, and is configured to perform various information processing based on programs and data. The storage unit 32 may be composed of, for example, a hard disk drive or a solid-state drive. In this embodiment, the storage unit 32 stores various information such as a control program 83 and learning result data 225.

[0086] The control program 83 is a program that causes the control device 3 to perform information processing (Figure 12, described later) regarding the detection of foreign object B using the trained detection model 5 and the retrieval of the detected foreign object B. The control program 83 includes a series of instructions for said information processing. At least one of the RAM 83 and the learning result data 225 may be stored in the storage medium 93. The control device 3 may also obtain at least one of the control program 83 and the learning result data 225 from the storage medium 93.

[0087] The control device 3 may communicate data with other computers (e.g., model generation device 2) via the communication interface 33. The control device 3 may be connected to the imaging device C2 and the retrieval device R via at least one of the communication interface 33 and the external interface 34. The control device 3 may be directly connected to at least one of the imaging device C2 and the retrieval device R, or indirectly connected via another computer. If the operation of the conveyor V is to be controllable, the control device 3 may be connected to the conveyor V via at least one of the communication interface 33 and the external interface 34. The control device 3 may accept operations and inputs from operators using the input device 35 and the output device 36. The operators of each device 1 to 3 may be at least partially the same, or they may not be the same.

[0088] Regarding the specific hardware configuration of the control device 3, components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control unit 31 may include multiple hardware processors. The hardware processors may consist of microprocessors, FPGAs, DSPs, GPUs, etc. The storage unit 32 may consist of RAM and ROM included in the control unit 31. At least one of the communication interface 33, external interface 34, input device 35, output device 36, and drive 37 may be omitted. The control device 3 may consist of multiple computers. In this case, the hardware configuration of each computer may or may not be the same. Furthermore, the control device 3 may be an information processing device designed specifically for the services provided, as well as a general-purpose server device, a general-purpose PC, an industrial PC, a terminal device, etc.

[0089] [Software Configuration] <Information Processing Device> Figure 5 schematically shows an example of the software configuration of the information processing device 1 according to this embodiment. The control unit 11 of the information processing device 1 loads the data acquisition program 81 stored in the storage unit 12 into the RAM. The control unit 11 then executes the instructions contained in the data acquisition program 81 loaded into the RAM. As a result, the information processing device 1 according to this embodiment operates as a computer equipped with an image acquisition unit 111, a recovery detection unit 112, and an image acquisition unit 113 as software modules. In other words, in this embodiment, each software module of the information processing device 1 is realized by the control unit 11 (CPU).

[0090] The image acquisition unit 111 is configured to continuously acquire captured images 121 of waste A being transported on the conveyor V by the imaging device C1. The recovery detection unit 112 is configured to detect when a worker P has manually recovered foreign object B using a detection device. In this embodiment, the recovery detection unit 112 is configured to detect when foreign object B has been recovered by performing image analysis on the captured images 121 acquired by the imaging device C1. The image collection unit 113 is configured to collect captured images 125 for machine learning when it is detected that a worker P has manually recovered foreign object B, from among the continuously acquired captured images 121, specifically those captured at the time when foreign object B is present within the imaging range of the imaging device C1.

[0091] <Model Generator 2> Figure 6 schematically shows an example of the software configuration of the model generation device 2 according to this embodiment. The control unit 21 of the model generation device 2 loads the model generation program 82 stored in the memory unit 22 into RAM. Then, the control unit 21 has the CPU execute the instructions contained in the model generation program 82 loaded into RAM. This enables the model generation according to this embodiment. Device 2 operates as a computer equipped with an image acquisition unit 211, a learning processing unit 212, and a storage processing unit 213 as software modules. In other words, in this embodiment, each software module of the model generation device 2 is also implemented by the control unit 21 (CPU), similar to the information processing device 1 described above.

[0092] The image acquisition unit 211 is configured to acquire multiple training images 221 relating to foreign object B among the waste A. At least a portion of the acquired training images 221 may be training images 125 collected by the information processing device 1. The learning processing unit 212 is configured to train the detection model 5 to acquire the ability to detect foreign object B in the input image by machine learning using the acquired training images 221. The storage processing unit 213 is configured to store the trained detection model 5 generated by machine learning.

[0093] In one example, the storage processing unit 213 may be configured to generate learning result data 225 that shows the trained detection model 5 generated as a result of machine learning. The structure of the learning result data 225 is not particularly limited and may be determined as appropriate depending on the embodiment, as long as it can hold information for performing computational processing of the trained detection model 5. For example, the learning result data 225 may be configured to include information showing the values ​​of computational parameters adjusted by machine learning. In some cases, the learning result data 225 may be configured to include information showing the structure of the trained detection model 5 (e.g., the structure of the neural network). The storage processing unit 213 may be configured to store the generated learning result data 225 in a predetermined memory area.

[0094] <Control device> Figure 7 schematically shows an example of the software configuration of the control device 3 according to this embodiment. The control unit 31 of the control device 3 loads the control program 83 stored in the storage unit 32 into the RAM. The control unit 31 then executes the instructions contained in the control program 83 loaded into the RAM using the CPU. As a result, the control device 3 according to this embodiment operates as a computer equipped with an image acquisition unit 311, a detection unit 312, and a retrieval processing unit 313 as software modules. In other words, in this embodiment, each software module of the control device 3 is also realized by the control unit 31 (CPU), similar to the information processing device 1 described above.

[0095] The image acquisition unit 311 is configured to acquire a target image 321 showing the waste A being transported on the conveyor V by the imaging device C2. The detection unit 312 holds the learning result data 225 and includes a trained detection model 5 generated by the model generation device 2. The detection unit 312 is configured to use the trained detection model 5 to detect whether or not foreign object B is present in the acquired target image 321. The recovery processing unit 313 is configured to drive the recovery device R to recover the foreign object B if, as a result of the detection, foreign object B is present in the acquired target image 321.

[0096] <Other> The software modules of the information processing device 1, the model generation device 2, and the control device 3 will be described in detail in the operation examples described later. In this embodiment, an example is described in which each software module of the information processing device 1, the model generation device 2, and the control device 3 is implemented by a general-purpose CPU. However, some or all of the above software modules may be implemented by one or more dedicated processors. Each of the above modules may also be implemented as a hardware module. Furthermore, regarding the software configuration of the information processing device 1, the model generation device 2, and the control device 3, software modules may be omitted, replaced, or added as appropriate, depending on the embodiment.

[0097] §3 Example of Operation [whole] Figure 8 is a flowchart showing an example of the overall system operation procedure according to this embodiment. However, the operation procedure described below is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following operation procedure may be omitted, replaced, or added as appropriate, depending on the embodiment.

[0098] In step S1, the conveyor V is driven. The conveyor V may be started manually, or it may be started at least partially automatically. In response to the conveyor V starting to drive, in step S2, the control device 3 of the foreign object recovery system 300 starts performing information processing related to foreign object recovery. Also, in step S3, the information processing device 1 of the data acquisition system 100 starts performing information processing related to the collection of training images 125. The control device 3 and the information processing device 1 may start the execution of each information processing automatically in conjunction with the starting of the conveyor V, or they may start the execution of each information processing manually.

[0099] In step S4, the conveyor V is stopped. The conveyor V may be stopped manually, or it may be stopped automatically, at least partially. In response to the conveyor V being stopped, the control device 3 and the information processing device 1 terminate the execution of each information processing. The control device 3 and the information processing device 1 may automatically terminate the execution of each information processing in conjunction with the stopping of the conveyor V, or they may terminate the execution of each information processing manually. Also, in step S5, the model generation device 2 performs information processing related to machine learning of the detection model 5. The model generation device 2 may automatically start the execution of information processing in conjunction with the stopping of the conveyor V, or it may start the execution of each information processing manually.

[0100] Once the trained detection model 5 is generated by the model generation device 2, the operation procedure of the system according to this embodiment is completed. Note that the cycle of steps S1 to S5 may be repeated. The timing of the repetition is not particularly limited and may be determined as appropriate depending on the embodiment.

[0101] [Information Processing Device] Figure 9 is a flowchart showing an example of the information processing procedure for collecting training images 125 in step S3 by the information processing device 1 according to this embodiment. The processing procedure of the information processing device 1 described below is an example of a data collection method. However, the processing procedure of the information processing device 1 described below is merely an example, and each step may be modified as much as possible. In addition, steps may be omitted, replaced, and added as appropriate in the following processing procedure, depending on the embodiment.

[0102] (Step S31) In step S31, the control unit 11 operates as an image acquisition unit 111 and continuously acquires captured images 121 of the waste A being transported on the conveyor V by the imaging device C1. The path and method for acquiring the captured images 121 may be appropriately selected depending on the embodiment. The control unit 11 may acquire the captured images 121 directly from the imaging device C1, or it may acquire the captured images 121 indirectly via another computer. Once the continuous acquisition of captured images 121 begins, the control unit 11 proceeds to the next step S32.

[0103] The processing from step S31 onward may be applied to each of the continuously obtained captured images 121. Furthermore, the control unit 11 may perform image analysis on the continuously obtained captured images 121 to determine whether or not the worker P's hand is visible in the captured image 121. That is, the control unit 11 may determine whether or not the worker P's hand is in the frame. The image analysis for detecting the worker P's hand may consist of, for example, general image processing such as pattern matching, or image processing using a trained machine learning model. Upon detecting that worker P's hand is visible, the control unit 11 may start recording the resulting captured image 121. The control unit 11 may then collect training images 125 from the recorded captured image 121 through subsequent processing. On the other hand, the control unit 11 may discard captured images 121 when worker P's hand is not visible. This allows for more efficient use of memory resources when collecting training images 125. In this case, the control unit 11 may continue recording the captured image 121 while worker P's hand is in the frame.

[0104] (Step S32) In step S32, the control unit 11 operates as a recovery detection unit 112 and detects that worker P has manually recovered the foreign object B using a detection device. In this embodiment, the control unit 11 detects that the foreign object B has been recovered by performing image analysis on the captured image 121 acquired by the imaging device C1.

[0105] Figure 10 is a flowchart showing an example of the processing procedure of a subroutine for recovering foreign matter B according to this embodiment. In this embodiment, the imaging device C1 may be positioned so as to include the work area of ​​worker P on the conveyor V within its imaging range. Accordingly, the processing of step S32 according to this embodiment may include the processing of steps S321 to S327 below. However, the processing procedure of the subroutine shown in Figure 10 is merely an example, and each process may be modified as much as possible. Furthermore, steps in the processing procedure shown in Figure 10 can be omitted, replaced, and added as appropriate, depending on the embodiment.

[0106] (Steps S321 and S322) In step S321, the control unit 11 detects whether or not worker P picked up an object in the work area by performing image analysis on the captured image 121 acquired by the imaging device C1. The image analysis in step S321 may consist of, for example, general image processing such as pattern matching, or image processing using a trained machine learning model. Note that in the image analysis in step S321, the control unit 11 does not need to recognize the object touched by worker P as a foreign object B.

[0107] In step S322, the control unit 11 determines the branch destination of the process according to the detection result of step S321. If the detection process determines that worker P's hand is touching the object ("YES" in step S322 of Figure 10), the control unit 11 proceeds to the next step S323. At this time, the control unit 11 may store the position and time on the image when worker P's hand first touched the object. On the other hand, if it determines that worker P's hand is not touching the object ("NO" in step S322 of Figure 10), the control unit 11 returns to step S321 and repeats the execution of the process from step S321.

[0108] (Steps S323 to S325) In step S323, after detecting that worker P has picked up an object, the control unit 11 continuously monitors the state in which worker P is holding the object by performing image analysis on the acquired captured image 121. The image analysis method may be the same as the detection method in step S321. In the process of step S323, the control unit 11 may determine whether or not worker P's hand is in continuous contact with the same object. The control unit 11 may also determine whether or not worker P has moved off the conveyor belt V while still in contact with the same object. Note that in the process of step S323, the control unit 11 does not need to recognize the object touched by worker P as a foreign object B.

[0109] In steps S324 and S325, the control unit 11 determines the branch destination of the process based on the results of the monitoring performed in step S323.

[0110] First, in step S324, the control unit 11 determines the branch destination of the process based on the result of determining whether or not the worker P's hand continues to be in contact with the same object on the conveyor V. The case in which the worker P's hand continues to be in contact with the same object on the conveyor V may correspond to a situation where the worker P is deciding whether to collect the object as foreign object B or return it to the conveyor V and not collect it. Alternatively, this case may correspond to a situation where the worker P's hand is overlapping with the object, causing it to be mistakenly detected as being in the worker P's hand. In other words, the case in which the worker P's hand continues to be in contact with the same object on the conveyor V may correspond to a situation where it has not been determined that the object is foreign object B. Therefore, if the control unit 11 determines that the worker P's hand continues to be in contact with the same object on the conveyor V ("YES" in step S324 of Figure 10), it returns to step S323 and continues to monitor the state in which the worker P has picked up the object.

[0111] On the other hand, the case in which worker P's hand no longer makes contact with an object on conveyor V corresponds to a situation where worker P has determined the object to be foreign object B and has moved foreign object B off conveyor V (i.e., worker P's hand is no longer on conveyor V, and therefore it is no longer determined that worker P is in contact with an object on conveyor V). Alternatively, this case may correspond to a situation where worker P has determined that the object is not foreign object B and has returned the object to conveyor V, or the false detection has been resolved (worker P's hand no longer overlaps the object). Therefore, when worker P's hand no longer makes contact with an object on conveyor V ("NO" in step S324 of Figure 10), the control unit 11 proceeds to the next step S325.

[0112] In step S325, the control unit 11 determines the branching point of the process based on the result of determining whether or not worker P's hand has moved to a distant position on the conveyor V. The case in which worker P's hand has moved to a distant position on the conveyor V corresponds to a situation in which worker P has determined the object to be a foreign object B and has carried the foreign object B off the conveyor V (in other words, worker P's hand has moved off the conveyor V while remaining in contact with the same object). Therefore, if worker P's hand has moved to a distant position on the conveyor V ("YES" in step S325 of Figure 10), the control unit 11 proceeds to step S326. That is, in this processing procedure, based on the result of monitoring in step S323, the control unit 11 can detect that worker P has moved the object off the conveyor V while holding it, in accordance with the determination of "NO" in step S324 and "YES" in step S325.

[0113] On the other hand, the case in which worker P's hand has not moved to a distant position on the conveyor V corresponds to a situation where worker P has determined that the object is not foreign object B and has returned the object to the conveyor V, or where the false detection has been resolved. In other words, this case may correspond to a situation where worker P's hand is on the conveyor V, but is no longer in contact with the same object (i.e., the object was not foreign object B). Therefore, if worker P's hand has not moved to a distant position on the conveyor V ("NO" in step S325 of Figure 10), the control unit 11 proceeds to step S327.

[0114] (Steps S326 and S327) In step S326, the control unit 11 recognizes that the object has been recovered as foreign object B, in response to worker P moving the object off the conveyor V while holding it in their hand. On the other hand, in step S327, the control unit 11 recognizes that foreign object B was not recovered. After making a decision regarding the recovery of foreign object B based on the processing in step S326 or step S327, the control unit 11 terminates the processing procedure of the subroutine in step S32 according to this embodiment and proceeds to step S33.

[0115] (Step S33) Returning to Figure 9, in step S33, the control unit 11 responds according to the detection result of step S32. The control unit 11 determines the branch destination for processing. If it detects in step S32 that worker P has recovered foreign object B, the control unit 11 proceeds to the next step S34. On the other hand, if it does not detect that worker P has recovered foreign object B (i.e., it recognizes that foreign object B has not been recovered), the control unit 11 omits the processing in step S34 and terminates the processing procedure of the information processing device 1 in this example of operation.

[0116] (Step S34) In step S34, the control unit 11 operates as an image acquisition unit 113 and collects the images captured at the time when the foreign object B is present within the imaging range of the imaging device C1 from the continuously acquired captured images 121 as training images 125 for machine learning.

[0117] The time at which foreign object B is likely to be present in the imaging range may be determined appropriately depending on the method for detecting the retrieval of foreign object B. The time at which foreign object B is likely to be present in the imaging range is the time before the time at which the retrieval of foreign object B is detected. In one example, the time before is set to a predetermined value. In another example, the time before is determined by any method. In this embodiment, the image captured to be collected as training image 125 may be an image captured at a time before the time when worker P picked up the object which is foreign object B. In one example, the control unit 11 may use the position and time at which the hand first touched the object in step S321 as a reference to determine which image to collect as training image 125 from the continuously acquired and recorded image capture images 121. The control unit 11 may collect one or more images from the recorded image capture images 121 as training images 125 in response to the detection of a single retrieval of foreign object B. The control unit 11 may store the collected training images 125 in any storage area.

[0118] The memory area may be, for example, RAM in the control unit 11, memory unit 12, external storage device, storage medium (storage medium 91, etc.), or a combination thereof. The storage medium may be, for example, a CD, DVD, etc., and the control unit 11 may store the training images 125 in the storage medium via the drive 17. The external storage device may be, for example, a data server such as a NAS (Network Attached Storage). In this case, the control unit 11 may use the communication interface 13 to store the training images 125 in the data server via the network. Alternatively, the external storage device may be, for example, an external storage device connected to the information processing device 1 via the external interface 14.

[0119] In one example, the captured image obtained by the imaging device C1 may be collected directly as the training image 125. In another example, the training image 125 may be obtained by applying arbitrary image processing to the captured image obtained by the imaging device C1. The arbitrary image processing may be, for example, a process to extract the portion containing the foreign object B. In this case, the control unit 11 may obtain the training image 125 by extracting the area in the captured image to be collected as the training image 125 in which the foreign object B is visible or is highly likely to be present, based on the position where the hand first touched the object in step S321. The image processing may be performed at least partially automatically, or it may be performed manually by an operator.

[0120] Furthermore, when used for supervised learning, the training images 125 may be assigned a correct label. The assignment of the correct label may be performed when collecting the training images 125. The correct label may be assigned automatically, at least partially, or it may be assigned manually. When assigning a correct label indicating the range of existence of foreign object B (bounding box, etc.), the range of existence of foreign object B may be estimated by computer processing based on the position where the hand first touched the object in step S321. The correct label may be automatically generated from the estimated range of existence. When performing image processing on the captured image to obtain the training image 125, information processing to assign the correct label may be performed at the same time as the image processing.

[0121] Once the training image 125 is collected, the control unit 11 terminates the processing procedure of the information processing device 1 according to this example of operation. The control unit 11 may repeatedly execute the series of information processing steps S31 to S34. In one example, the control unit 11 may continuously and repeatedly execute the series of information processing steps S31 to S34 while the conveyor V is being driven. This allows the training image 125 of the foreign object B collected by the worker P to be collected while the waste A is being transported by the conveyor V. The timing of the repetition is not particularly limited and may be determined as appropriate depending on the embodiment.

[0122] [Model Generator] Figure 11 is a flowchart showing an example of the information processing procedure for machine learning of the detection model 5 in step S5 by the model generation device 2 according to this embodiment. The processing procedure of the model generation device 2 described below is an example of a model generation method. However, the processing procedure of the model generation device 2 described below is merely an example, and each step may be modified as much as possible. In addition, steps may be omitted, replaced, and added as appropriate in the following processing procedure, depending on the embodiment.

[0123] (Step S51) In step S51, the control unit 21 operates as an image acquisition unit 211 and acquires multiple training images 221 related to foreign objects B among the waste A. The number of training images 221 to be acquired may be determined as appropriate depending on the embodiment.

[0124] Training images 221 may be generated as appropriate. For example, training images 221 may be generated by preparing a foreign object and imaging the conveying surface of the conveyor V with or without the foreign object using an imaging device. Training images 221 may be generated automatically by the operation of a computer, or they may be generated manually, including at least partly operator operation. The generated training images 221 may be stored on another computer, such as a network server. When performing machine learning, the control unit 21 may acquire training images 221 via a network, external storage device, storage medium 92, etc.

[0125] At least a portion of the training images 221 may be generated by the model generation device 2. At least a portion of the training images 221 may be generated by a computer other than the model generation device 2 (information processing device 1 or another computer). At least a portion of the multiple training images 221 may be training images 125 obtained by the information processing of the data collection described above. If the training images 221 are generated by another computer, the control unit 21 may acquire the training images 221 generated by the other computer via, for example, a network, external storage device, storage medium 92, etc.

[0126] As a preprocessing step S51, at least one of the above-mentioned assignment of correct labels and image processing may be performed. The preprocessing may be performed by the information processing device 1, the model generation device 2, or another computer. If unsupervised learning is adopted as the form of machine learning, the assignment of correct labels may be omitted. Once the training images 221 are acquired, the control unit 21 proceeds to the next step S52.

[0127] (Step S52) In step S52, the control unit 21 trains the detection model 5 to acquire the ability to detect foreign objects B in the input image by machine learning using the acquired training images 221.

[0128] As described above, training the detection model 5 involves deriving an output (inference result) that fits the training image 221 from the training image 221 by adjusting the values ​​of the computational parameters included in the detection model 5. The detection model 5 may be configured by adjusting (optimizing) it. The type, configuration, and structure of the detection model 5 are not particularly limited and may be appropriately selected depending on the embodiment. The method for adjusting the computation parameters in machine learning may be appropriately selected depending on the type and configuration of the machine learning model to be adopted. For example, the adjustment method may be the backpropagation method or solving an optimization problem. When the trained detection model 5 is generated as a result of performing machine learning, the control unit 21 proceeds to the next step S53.

[0129] (Step S53) In step S53, the control unit 21 operates as a storage processing unit 213 and generates learning result data 225 containing information about the trained detection model 5 generated by machine learning. The learning result data 225 may be configured as appropriate to include information for reconstructing the trained detection model 5. The control unit 21 stores the generated learning result data 225 in a predetermined memory area.

[0130] The predetermined memory area may be, for example, RAM in the control unit 21, memory unit 22, external storage device, storage media, or a combination thereof. The storage media may be, for example, a CD, DVD, etc., and the control unit 21 may store the learning result data 225 in the storage media via the drive 27. The external storage device may be, for example, a data server such as a NAS. In this case, the control unit 21 may use the communication interface 23 to store the learning result data 225 in the data server via the network. Alternatively, the external storage device may be, for example, an external storage device connected to the model generation device 2 via the external interface 24.

[0131] Once the saving of the machine learning results is complete, the control unit 21 terminates the processing procedure of the model generation device 2 in this example of operation.

[0132] The generated learning result data 225 may be provided to the control device 3 at any timing and in any manner. For example, the control unit 21 may transfer the learning result data 225 to the control device 3 as part of the processing in step S53 or separately from the processing in step S53. The control device 3 may acquire the learning result data 225 by receiving this transfer. Alternatively, for example, the control device 3 may acquire the learning result data 225 by accessing the model generation device 2 or data server via a network using the communication interface 33. Alternatively, for example, the control device 3 may acquire the learning result data 225 via the storage medium 93. Initially, the learning result data 225 may be pre-loaded into the control device 3.

[0133] Furthermore, the control unit 21 may update or generate new learning result data 225 by repeatedly executing the processes in steps S51 to S53 periodically or irregularly. During this repetition, at least a portion of the training images 221 used for machine learning may be changed, modified, added, or deleted as appropriate. If new training images 125 are obtained by the data collection system 100, the newly obtained training images 125 may be preferentially used as training images 221 for machine learning. The control unit 21 may then update the learning result data 225 held by the control device 3 by providing the updated or newly generated learning result data 225 to the control device 3 in any way.

[0134] [Control device] Figure 12 is a flowchart showing an example of the information processing procedure for foreign object recovery in step S2 by the control device 3 according to this embodiment. The processing procedure of the control device 3 described below is an example of a control method. However, the processing procedure of the control device 3 described below is merely an example, and each step may be modified as much as possible. In addition, steps may be omitted, replaced, and added as appropriate in the following processing procedure, depending on the embodiment.

[0135] (Step S21) In step S21, the control unit 31 operates as an image acquisition unit 311 and acquires a target image 321 showing the waste A being transported on the conveyor V by the imaging device C2. The control unit 31 may acquire the target image 321 directly from the imaging device C2, or it may acquire the target image 321 indirectly via another computer or the like. Once the target image 321 is acquired, the control unit 31 proceeds to the next step S22.

[0136] (Step S22) In step S22, the control unit 31 operates as a detection unit 312 and uses the trained detection model 5 to detect whether or not a foreign object B is present in the acquired target image 321.

[0137] The computational processing of the trained detection model 5 may be appropriately determined depending on the type, configuration, structure, etc. of the detection model 5. In one example, if the detection model 5 is composed of a neural network, the control unit 31 inputs the target image 321 to the trained detection model 5 and performs forward propagation computation of the trained detection model 5. As a result of performing this computation, the control unit 31 can obtain an output value from the trained detection model 5 that corresponds to the result of detecting whether or not a foreign object B is present in the target image 321.

[0138] Furthermore, at any time before executing the process in step S22, the control unit 31 may refer to the learning result data 225 and set the trained detection model 5 to a state where it can perform calculation processing. The control unit 31 may also perform any preprocessing on the acquired target image 321. The control unit 31 may input the data obtained by the preprocessing as the target image 321 into the trained detection model 5 and execute the calculation processing of the trained detection model 5. Once the process of detecting foreign object B is complete, the control unit 31 proceeds to the next step S23.

[0139] (Step S23) In step S23, the control unit 31 determines the branch destination of the process according to the detection result in step S22. If, as a result of the detection in step S22, foreign object B is present in the acquired target image 321, the control unit 31 proceeds to the next step S24. On the other hand, if foreign object B is not present in the acquired target image 321, the control unit 31 omits the processing in the next step S24 and terminates the processing procedure of the control device 3 in this example of operation.

[0140] (Step S24) In step S24, the control unit 31 operates as a recovery processing unit 313 and drives the recovery device R to recover the foreign object B.

[0141] The method for driving the recovery device R for recovering foreign object B may be determined as appropriate depending on the embodiment. In one example, the recovery device R may be a robotic device. In this case, the control unit 31 may provide the recovery device R with positional information of foreign object B as it appears in the target image 321, thereby causing the recovery device R to recover the foreign object B. In another example, the recovery device R may be composed of a recovery mechanism such as a trap. In this case, the control unit 31 may drive the recovery device R at the timing when the foreign object B that appears in the target image 321 passes through the recovery device R, thereby causing the recovery device R to recover the foreign object B.

[0142] Once the information processing for recovering foreign matter B is completed, the control unit 31 terminates the processing procedure of the control device 3 in this example of operation. The control unit 31 may repeatedly execute the series of information processing steps S21 to S24. In one example, the control unit 31 may continuously and repeatedly execute the series of information processing steps S21 to S24 while the conveyor V is being driven. This allows the foreign matter to be recovered by the recovery device R while waste A is being transported by the conveyor V. The recovery of B can be carried out continuously. The timing of the repetition is not particularly limited and may be determined as appropriate depending on the embodiment.

[0143] [Features] In this embodiment, the process described in steps S31 to S34 enables the automation of collecting training images 125 containing foreign object B, triggered by the manual collection of foreign object B. This reduces the cost of collecting training images of waste collected as foreign objects at the site. Furthermore, in steps S51 to S53, at least a portion of the collected training images 125 can be used for machine learning to generate a trained detection model 5 that has acquired the ability to detect the foreign object. In steps S21 to S24, the generated trained detection model 5 can be used to automate at least a portion of the work of detecting foreign objects at the site and collecting the detected foreign objects. Therefore, the data collection system 100 according to this embodiment can reduce the cost of collecting training images to generate a trained machine learning model with such capabilities. As a result, the system according to this embodiment can reduce the cost of collecting foreign objects from waste.

[0144] §4 Variant Although embodiments of the present invention have been described in detail above, the above description is merely illustrative in all respects of the present invention. Needless to say, various improvements or modifications can be made without departing from the scope of the present invention. For example, the following modifications are possible. In the following, the same reference numerals are used for components similar to those in the above embodiments, and explanations of points similar to those in the above embodiments have been omitted as appropriate. The following modifications can be combined as appropriate.

[0145] <4.1> In the above embodiment, at least one of the data acquisition system 100, the model generation device 2, and the foreign object recovery system 300 may be omitted.

[0146] Furthermore, in the above embodiment, the model generation device 2 may generate the trained detection model 5 at any time other than those described above. If the training image 125 has not yet been obtained, the model generation device 2 may generate the trained detection model 5 using a plurality of training images that do not include the training image 125. The generated trained detection model 5 may be used in the processing of step S22 of the foreign object recovery system 300.

[0147] <4.2> In the above embodiment, the detection device is comprised of an imaging device C1. However, the configuration of the detection device is not limited to this example. As long as it can detect the retrieval of foreign object B by worker P, the configuration of the detection device may be appropriately determined according to the embodiment. In another example, the retrieval of foreign object B by worker P may be performed by placing foreign object B into a retrieval box RB. Accordingly, the detection device may be comprised of a sensor that senses that foreign object B has been placed into the retrieval box RB.

[0148] Figure 13 schematically shows an example of another scenario in which the present invention is applied. In the modified example shown in Figure 13, the detection device is comprised of a sensor S. The sensor S is configured to sense that a foreign object B has been placed in the collection box RB. The type of sensor S is not particularly limited and may be appropriately selected depending on the embodiment. The sensor S may be comprised of, for example, an infrared sensor or other sensor that detects the passage of an object. In this case, the sensor S may be placed near the opening (inlet) of the collection box RB and may sense whether or not an object has passed through the opening (i.e., has been placed in from the outside).

[0149] In this modified example, the control unit 11 of the information processing device 1 detects whether or not worker P has manually retrieved the foreign object B, based on the sensing result of the sensor S, instead of performing the process in step S32 above. For example, when the sensor S senses that an object (foreign object B) has been placed in the collection box RB, the control unit 11 detects that worker P has manually retrieved the foreign object B. On the other hand, as long as the sensor S does not sense that an object has been placed in the collection box RB, the control unit 11 determines that the foreign object B has not been retrieved.

[0150] In step S34 described above, the control unit 11, in response to detecting the recovery of foreign object B based on the sensing result of the sensor S, collects the image acquired at the time when the foreign object B is likely to be present in the imaging range of the imaging device C1 from the continuously acquired image 121 as training image 125. The time when the foreign object B is likely to be present in the imaging range may be a time prior to the time when the recovery of foreign object B is detected by the sensor S. In one example, the time prior may be set to a predetermined value. In another example, the time prior may be determined by any method. With respect to other configurations, the system according to this modified example may be the same as in the embodiment described above.

[0151] According to this modified example, the sensing results from sensor S are used to detect the recovery of foreign object B by worker P. This simplifies the information processing required to detect the recovery of foreign object B.

[0152] In another example, worker P may wear gloves to retrieve the foreign object B. These gloves may be equipped with a sensor (e.g., a load sensor) to detect whether or not an object is being grasped. The detection device may consist of this sensor. In yet another example, an RFID tag may be attached to worker P's hand or glove, and a circuit capable of communicating with the RFID tag may be attached to the retrieval box RB. In this case, the detection device may consist of the RFID tag. In response to the RFID tag communicating with the circuit in the retrieval box RB, it may be detected that worker P's hand has moved to the retrieval box RB, i.e., that the foreign object B has been retrieved. In yet another example, a weight-detecting sensor, such as a load sensor, may be attached to the retrieval box RB. The detection device may consist of this sensor. In this case, the retrieval of the foreign object B may be detected based on the weight measured by the sensor (e.g., in response to an increase in the weight of the object placed in the retrieval box RB). [Explanation of Symbols]

[0153] 100...Data collection system, 1... Information processing device, C1... Imaging device, 11...Control unit, 12...Storage unit, 13...Communication interface, 14...External interface, 15...Input device, 16...Output device, 17...Drive, 81...Data acquisition program, 91...Storage medium, 111...Image acquisition unit, 112...Recovery and detection unit, 113...Image Collection Department, 121... Captured image, 125... Training image, 2...Model generation device, 21...Control unit, 22...Storage unit, 23...Communication interface, 24...External interface, 25...Input device, 26...Output device, 27...Drive, 82...Model generation program, 92...Storage medium, 211...Image acquisition unit, 212...Learning processing unit, 213... Storage and Processing Unit, 221...Training images, 225...Learning result data, 300... Foreign object recovery system, 3...Control device, C2...Imaging device, R...Recovery device, 31...Control unit, 32...Storage unit, 33...Communication interface, 34...External interface, 35...Input device, 36...Output device, 37...Drive, 83...Control program, 93...Storage medium, 311...Image acquisition unit, 312...Detection unit, 313... Recovery Processing Unit, 321...Target image, 5...Detection model, A... Garbage, B... Foreign object, V...Conveyor, RB...Collection box

Claims

1. An imaging device positioned to image waste being transported on a conveyor belt. A detection device for detecting that foreign objects among the aforementioned waste have been collected manually by a worker, and Information processing device, Equipped with, The aforementioned information processing device is The imaging device continuously acquires images of the waste being transported on the conveyor. The detection device detects that the worker has manually collected the foreign object, and When it is detected that the worker has manually retrieved the foreign object, the image captured at a time prior to the time the worker manually retrieved the foreign object, while the foreign object was still within the imaging range of the imaging device, is collected from the continuously acquired images as training images for machine learning. Configured to perform, Data collection system.

2. The detection device is comprised of the imaging device, The detection of the recovery of the foreign object by the detection device is performed by analyzing the captured image acquired by the imaging device to detect that the foreign object has been recovered. The data acquisition system according to claim 1.

3. The imaging device is positioned so as to include the worker's work area on the conveyor belt within its imaging range. Detecting that the aforementioned foreign object has been recovered means By performing image analysis on the captured images acquired by the aforementioned imaging device, it is possible to detect that the worker picked up an object in the work area. After detecting that the worker has picked up the object, the state in which the worker is holding the object is continuously monitored by performing image analysis on the acquired captured image. Based on the monitoring results, it is detected that the worker moved the object off the conveyor belt while holding the object in their hand, and The system detects that the object has been recovered as a foreign object when the worker moves the object off the conveyor belt while holding it in their hand. It is composed of, The captured images collected as training images are images taken at a time prior to the time the worker picked up the foreign object. The data acquisition system according to claim 2.

4. The foreign matter is collected by placing it into a collection box. The detection device consists of a sensor that senses when the foreign object has been placed in the collection box. The data acquisition system according to claim 1.

5. A data collection method performed in a data collection system comprising an imaging device positioned to image waste being transported on a conveyor belt, a detection device for detecting when foreign objects among the waste have been manually collected by a worker, and an information processing device, The aforementioned information processing device The imaging device continuously acquires images of the waste being transported on the conveyor. The detection device detects that the worker has manually collected the foreign object, and When it is detected that the worker has manually retrieved the foreign object, the image captured at a time prior to the time the worker manually retrieved the foreign object, while the foreign object was still within the imaging range of the imaging device, is collected from the continuously acquired images as training images for machine learning. including, Data collection methods.

6. The detection device is comprised of the imaging device, The detection of the recovery of the foreign object by the detection device is performed by analyzing the captured image acquired by the imaging device to detect that the foreign object has been recovered. The data acquisition method according to claim 5.

7. The imaging device is positioned so as to include the worker's work area on the conveyor belt within its imaging range. Detecting that the aforementioned foreign object has been recovered means By performing image analysis on the captured images acquired by the aforementioned imaging device, it is possible to detect that the worker picked up an object in the work area. After detecting that the worker has picked up the object, the state in which the worker is holding the object is continuously monitored by performing image analysis on the acquired captured image. Based on the monitoring results, it is detected that the worker moved the object off the conveyor belt while holding the object in their hand, and The system detects that the object has been recovered as a foreign object when the worker moves the object off the conveyor belt while holding it in their hand. It is composed of, The captured images collected as training images are images taken at a time prior to the time the worker picked up the foreign object. The data collection method according to claim 6.

8. Computers To obtain multiple training images of foreign objects in the waste, The detection model is trained to acquire the ability to detect the foreign object in the input image by machine learning using the acquired multiple training images, and To store the trained detection model generated by the aforementioned machine learning, A model generation method that performs the following: At least a portion of the acquired training images are training images collected by the data acquisition system described in any one of claims 1 to 4. Model generation method.

9. An imaging device positioned to image waste being transported on a conveyor belt. Recovery device, and control device, Equipped with, The control device is The imaging device acquires an image of the waste being transported on the conveyor. Using the trained detection model generated by the model generation method described in claim 8, detect whether or not a foreign object is present in the acquired target image. If the detection results indicate that the foreign object is present in the acquired target image, the recovery device is driven to recover the foreign object. Configured to perform, Foreign object retrieval system.

Images