Determination system, determination method, and computer program

Abstract

This determination system comprises: an imaging device provided in a conveyance machine for conveying iron scrap; and a foreign matter determination device that acquires a determination target image captured by the imaging device and determines whether the iron scrap includes foreign matter.

Description

Determination system, determination method, and computer program

[0001] The present invention relates to a determination system, a determination method, and a computer program. This application claims priority based on Japanese Patent Application No. 2023-190706, filed on November 8, 2023, the contents of which are incorporated herein by reference.

[0002] In recent years, there has been an increasing demand in the steel industry for the recycling of steel, with the aim of reducing carbon dioxide emissions and other benefits. There are multiple types of scrap (steel scrap) that can be recycled. For example, scrap generated during the processing of steel products is classified as high-grade scrap, while H2 generated during the demolition of buildings is mainly classified as low-grade scrap. To produce high-grade steel, it is desirable to use high-grade high-grade scrap, but using only high-grade scrap could lead to future supply shortages. Therefore, it is necessary to utilize low-grade scrap as well.

[0003] Low-grade scrap contains a high percentage of impurities other than iron, and the content varies widely. This is largely due to the presence of prohibited materials, such as motors and electrical distribution boards, in the scrap. Among these impurities, elements that are difficult to remove during the steelmaking process (tramp elements: TE, representative elements: Cu, Sn, Ni, Cr, and Mo) can cause cracks during hot rolling and lead to poor material (mechanical properties). To address this issue, Patent Document 1 discloses a technology that uses an overhead crane as a concrete example of a scrap iron transport device to detect prohibited materials along its path. Specifically, the technology is as follows: An overhead crane is installed in advance, and the scrap iron is transported by the overhead crane. The overhead crane's travel path is fixed in advance, and a camera is installed along the path. As the scrap iron is transported by the overhead crane, an image of the scrap iron is captured by the camera along the path. Images captured by the camera are used to determine the presence of prohibited materials and prompt the worker to remove them.

[0004] Japanese Patent Application Laid-Open No. 2020-176909

[0005] As mentioned above, conventional technology is based on a system in which the movement path of the iron scrap conveying device is fixed. On the other hand, heavy machinery can be used to transport iron scrap. Purchasing heavy machinery has the advantage of reducing the initial investment compared to constructing an overhead crane. Furthermore, while an overhead crane has a fixed movement path, heavy machinery has the advantage of being free to move around, meaning that the iron scrap transport path is not fixed. Because it can move freely, it can be unloaded or loaded at any location within the iron scrap storage area. Therefore, compared to an overhead crane, which limits the location of the iron scrap storage area to locations accessible by the overhead crane, heavy machinery offers significantly greater flexibility in the use of the iron scrap storage area.

[0006] However, conventional methods of fixing cameras to fixed objects such as the ground or walls necessitate the installation of numerous fixed cameras throughout the entire operating area of ​​heavy machinery, which has become increasingly widespread as the flexibility of transport routes has increased. This increases the cost of installing cameras. To reduce the number of camera installation locations, it is possible to define the transport route of heavy machinery so that scrap iron passes close to the cameras. However, maintaining a constant distance between the camera and the scrap iron is not easy, as it increases the burden on the heavy machinery operator. This can lead to the distance between the scrap iron and the camera changing with each image capture, potentially reducing detection accuracy. Furthermore, the flexibility of the transport route may be reduced. The aforementioned issues associated with fixing cameras to fixed objects are also present in conventional techniques using overhead cranes. Furthermore, these issues are not limited to the detection of prohibited substances containing tramp elements, but are common to all foreign objects that must be detected for other reasons.

[0007] Therefore, the present invention has been made in consideration of the above-mentioned circumstances, and provides a technology that makes it possible to accurately determine whether or not foreign matter is mixed in iron scrap when a conveying machine is used to transport the iron scrap.

[0008] (1) One aspect of the present invention is an evaluation system that includes an imaging device provided on a conveying machine for conveying iron scrap, and a foreign matter evaluation device that acquires an image of an object to be evaluated captured by the imaging device and determines whether the iron scrap contains foreign matter.

[0009] (2) In one aspect of the present invention, in the determination system described in (1) above, the imaging device is provided on a main body or a movable body of the conveying machine.

[0010] (3) In one aspect of the present invention, in the determination system described in (1) or (2) above, the imaging device photographs the iron scrap during the process of being transported by the transport machine.

[0011] (4) In one aspect of the present invention, the determination system according to any one of (1) to (3) above further comprises a timing determination device that determines a processing timing, which is any one of the image acquisition timing for acquiring the determination target image, the imaging timing for imaging the determination target image, and the output timing for outputting a determination result by the foreign matter determination device. When the image acquisition timing is determined by the timing determination device, the foreign matter determination device acquires an image captured at the image acquisition timing from among images captured by the imaging device. When the imaging timing is determined by the timing determination device, the imaging device captures an image at the imaging timing. When the output timing is determined by the timing determination device, the foreign matter determination device outputs a determination result made at the output timing.

[0012] (5) In one aspect of the present invention, in the determination system described in (4) above, the timing determination device determines the processing timing based on an image captured by the imaging device or control information of the transport machine.

[0013] (6) In one aspect of the present invention, in the judgment system described in any one of (1) to (5) above, the foreign matter judgment device acquires an image of the object to be judged, which is taken from below, of the iron scrap being lifted by the conveyor, and makes a judgment.

[0014] (7) In one aspect of the present invention, in the judgment system described in any one of (1) to (6) above, the foreign matter judgment device acquires an image of the object to be judged, which is captured while the holding portion of the iron scrap being lifted by the conveyor is facing the imaging device, and makes a judgment.

[0015] (8) In one aspect of the present invention, in the judgment system described in any one of (1) to (7) above, the foreign matter judgment device acquires an image of the object to be judged, which is taken when the iron scrap has been lifted at least once by the conveying machine and placed in an area different from the destination before being transported to the specified destination, and makes a judgment.

[0016] (9) One aspect of the present invention is a determination method including an image capturing step of capturing an image using an imaging device provided on a conveying machine for transporting iron scrap, an acquisition step of acquiring the image captured by the imaging device as an image to be determined, and a foreign matter determination step of determining whether the iron scrap contains foreign matter using the image to be determined acquired in the acquisition step.

[0017] (10) In one aspect of the present invention, in the determination method described in (9) above, in the image capturing step, an image is captured from below of the iron scrap being lifted by the conveyor.

[0018] (11) In one aspect of the present invention, in the determination method described in (9) or (10) above, in the image capturing step, an image is captured with the holding portion of the iron scrap being lifted by the conveying machine facing the imaging device.

[0019] (12) In one aspect of the present invention, in the determination method described in any one of (9) to (11) above, in the image capturing step, an image is captured of the iron scrap that has been lifted at least once by the conveying machine and placed in an area different from the destination before being transported to the specified destination.

[0020] (13) One aspect of the present invention is a computer program for causing a computer to function as a foreign matter determination device that acquires an image of an object to be determined captured by an imaging device provided on a conveying machine for conveying iron scrap, and determines whether the iron scrap contains a foreign matter using the acquired image of the object to be determined.

[0021] According to the present invention, even when a conveying machine is used to convey the iron scrap to be recycled, it is possible to accurately determine whether or not foreign matter is present in the iron scrap.

[0022] 1 is a schematic block diagram showing the system configuration of a determination system 100 according to a first embodiment of the present invention. FIG. 1 is a diagram showing an example of a display output to a user. FIG. 2 is a diagram showing an application example of the determination system 100. FIG. 3 is a diagram showing variations in the installation mode of the imaging device 10. FIG. 4 is a schematic block diagram showing a specific example of the functional configuration of a foreign object determination device 20. FIG. 5 is a diagram showing a specific example of an image of iron scrap used as training data. A flowchart showing a specific example of processing by the foreign object determination device 20. FIG. 6 is a diagram showing a first specific example of implementation of the determination system 100 according to the first embodiment. FIG. 7 is a diagram showing a second specific example of implementation of the determination system 100 according to the first embodiment. FIG. 8 is a diagram showing a third specific example of implementation of the determination system 100 according to the first embodiment. FIG. 9 is a diagram showing a fourth specific example of implementation of the determination system 100 according to the first embodiment. FIG. 10 is a diagram showing a specific example of the state of heavy equipment 40. FIG. 11 is a diagram showing an overview of area C. FIG. 11 is a schematic block diagram showing the system configuration of a determination system 100 according to a second embodiment of the present invention. FIG. 12 is a schematic block diagram showing a specific example of the functional configuration of a timing determination device 50. FIG. 13 is a diagram showing a specific example of training data used for performing learning processing of a timing determination model. 1 is a flowchart showing a specific example of processing by the determination system 100 in the second embodiment. FIG. 2 is a diagram showing a first specific example of implementation of the determination system 100 in the second embodiment. FIG. 3 is a diagram showing a second specific example of implementation of the determination system 100 in the second embodiment. FIG. 4 is a diagram showing a third specific example of implementation of the determination system 100 in the second embodiment. FIG. 5 is a diagram showing a fourth specific example of implementation of the determination system 100 in the second embodiment. FIG. 6 is a diagram showing a fifth specific example of implementation of the determination system 100 in the second embodiment. FIG. 7 is a diagram showing an outline of an example of the hardware configuration of an information processing device 90 applied to this embodiment. FIG. 8 is a flowchart showing a first modified example of processing by the determination system 100 in the second embodiment. FIG. 9 is a flowchart showing a second modified example of processing by the determination system 100 in the second embodiment.

[0023] First Embodiment Fig. 1 is a schematic block diagram showing the system configuration of a judging system 100 according to a first embodiment of the present invention. The judging system 100 is used to judge the presence or absence of foreign matter contained in a steel scrap to be judged, based on an image of the steel scrap to be judged (hereinafter referred to as a "judgment target image") captured by an imaging device 10. Note that in this embodiment, "capturing" refers to the generation of electronic image data in response to light entering the imaging element of the imaging device 10. In other words, "capturing" is not limited to the recording of image data on a non-volatile recording medium.

[0024] The determination system 100 includes an imaging device 10, a foreign object determination device 20, and an output device 30. The imaging device 10 and the foreign object determination device 20 are connected so that at least image data captured by the imaging device 10 can be output to the foreign object determination device 20. The imaging device 10 and the foreign object determination device 20 may be integrally configured, for example, so that image data can be input and output via a bus. The imaging device 10 and the foreign object determination device 20 may be connected, for example, by a cable, so that image data can be input and output. The imaging device 10 and the foreign object determination device 20 may input and output image data by, for example, short-range wireless communication, or by communicating via a network (for example, a LAN, a mobile communication network, or the Internet). Note that in this embodiment, "acquiring an image" refers to the foreign object determination device acquiring image data generated by imaging.

[0025] The foreign object determination device 20 and the output device 30 are connected so as to be able to output, to the output device 30, at least, determination result data indicating the determination result by the foreign object determination device 20. The foreign object determination device 20 and the output device 30 may be configured as an integrated unit, for example, so that the determination result data can be input and output via a bus. The foreign object determination device 20 and the output device 30 may be connected by, for example, a cable, so that the determination result data can be input and output. The foreign object determination device 20 and the output device 30 may input and output the determination result data by, for example, short-range wireless communication, or may input and output the determination result data by communicating via a network (for example, a LAN, a mobile communication network, or the Internet).

[0026] The imaging device 10 is a device equipped with at least an imaging element and configured to capture images of the iron scrap to be assessed. The captured images may be still images or moving images. The imaging device 10 is configured using imaging devices such as a digital still camera, a digital video camera, a smartphone, a tablet, or a smart camera. The imaging device 10 outputs data of the captured images (particularly the images to be assessed) to the foreign object assessment device 20. Note that multiple imaging devices 10 may be installed in one assessment system 100. For example, multiple imaging devices 10 may be installed for one heavy equipment 40, which will be described later. More specifically, multiple imaging devices 10 may be installed for the heavy equipment 40 at positions and orientations illustrated in FIGS. 13 to 15, which will be described later. For example, an imaging device 10 positioned and oriented as shown in FIG. 13, an imaging device 10 positioned and oriented as shown in FIG. 14, and an imaging device 10 positioned and oriented as shown in FIG. 15 may each be installed on the same heavy equipment 40. Note that the heavy equipment 40 is a specific example of a conveyor.

[0027] The foreign matter determination device 20 determines whether or not the iron scrap to be determined contains a predetermined foreign matter based on the determination target image (the determination target image including the iron scrap) captured by the imaging device 10. Specific examples of the predetermined foreign matter to be determined include objects such as those listed in (1) to (4). (1) An object containing a large amount of elemental components (tramp elements) that are difficult to remove in the steelmaking process. (2) An enclosed object (e.g., a cylinder or tank) that poses a risk of explosion in the steelmaking process (e.g., the melting process). (3) A large or high-strength object that poses a risk of damaging the shredder or other crusher that feeds the iron scrap into the crusher. (4) A combustible object that may cause a fire or the like in the steelmaking process. The foreign matter determination device 20 outputs data indicating the determination result (determination result data) to the output device 30.

[0028] The output device 30 performs output processing in accordance with the determination result of the foreign object determination device 20. The output device 30 outputs information in a format recognizable by a user (e.g., a worker using the determination system 100). The output device 30 may be configured, for example, using an image output device (e.g., a display) capable of displaying text and images, or a device (e.g., a speaker or headphones) capable of outputting sound and voice. The output device 30 may also be configured using an information device to which these devices can be connected, or an information device that incorporates these devices. Specific examples of such information devices include smartphones, tablets, and dedicated devices. FIG. 2 is a diagram showing an example of a display output to a user. A motor, which is a foreign object, is detected among the iron scrap held by the lift magnet, and a rectangle is generated at the position of the motor to surround it. As shown in FIG. 2, the confidence level of the foreign object determination model for the detection result (0.93 in FIG. 2; minimum value is 0.0, maximum value is 1.0) may also be displayed.

[0029] FIG. 3 is a diagram illustrating an application example of the determination system 100. As shown in FIG. 3, the determination system 100 may be applied to a mobile heavy machine 40. In the example of FIG. 3, the heavy machine 40 is a device capable of transporting iron scrap. The heavy machine 40 may also be a device capable of transporting iron scrap over longer distances by self-propelling. The heavy machine 40 may be, for example, a hydraulic excavator. The heavy machine 40 includes, for example, a main body 403, a movable body 401 movable relative to the main body 403, a running body 406 that propels the heavy machine 40, and a connecting device, such as a swivel device, that displaceably connects the main body 403 and the like to the running body 406. In the example of FIG. 3, a lift magnet 402 is attached to the tip of the movable body 401 of the heavy machine 40 as a transport attachment (transport tool). The movable body 401 may include, for example, a boom 401a and an arm 401b, or may include movable parts other than the boom 401a and the arm 401b. The attachment attached to the tip of the movable body 401 does not have to be limited to the lift magnet 402. The attachment attached to the tip of the movable body 401 may have any configuration as long as it is capable of holding the iron scrap in the air. For example, an attachment that can hold (grasp) the iron scrap by opening and closing multiple movable members, such as a grapple or a crocodile, may be used. The main body 403 includes, for example, a cockpit 404 that forms a space for the operator to ride in, and a rotating body 405 (displacing body) that rotates (displaces) together with the cockpit 404.

[0030] The heavy equipment 40 lifts the iron scrap located in area A and moves it to area B. The heavy equipment 40 may operate in response to operation by an operator or control by a predetermined information processing device 90. In the example of FIG. 3 , the heavy equipment 40 is provided with an imaging device 10, a foreign object determination device 20, and an output device 30. The imaging device 10 may be attached to, for example, the main body 403 of the heavy equipment 40 (e.g., the cockpit 404 or the rotating body 405 near the cockpit 404) and installed facing a predetermined direction (e.g., a direction toward the tip of the movable body 401). Specifically, the imaging device 10 may be installed facing, for example, forward and diagonally downward from the main body 403. Areas A and B may each be areas of any type. For example, one or both of area A and area B may be a flat surface such as the ground or a floor, a loading platform of a truck or a railroad freight car, a hold of a ship, a container for transporting scrap iron (e.g., a pot-shaped container), an inlet for feeding scrap iron into a device for processing the scrap iron (e.g., a scrap chute), or other area. As a specific example, area A may be a loading platform, and area B may be the ground designated as a storage area for scrap iron.

[0031] 4 and 5 are diagrams showing variations in the installation mode of the imaging device 10. The imaging device 10 may be installed in any position in the heavy equipment 40 as long as it can capture an image of the iron scrap to be evaluated. As shown in FIG. 4 , the imaging device 10 may be installed at a relatively low position in the movable body 401 (e.g., a position relatively close to the cockpit 404). In this case, the imaging device 10 may be installed facing diagonally downward. As shown in FIG. 5 , the imaging device 10 may be installed at a relatively high position in the movable body 401 (e.g., a position relatively close to a joint portion of the movable body 401). In this case, the imaging device 10 may be installed facing diagonally downward or directly downward. When the imaging device 10 is installed facing diagonally downward in the mode shown in FIG. 5 , the imaging device 10 may be installed at an angle closer to a directly downward direction than in the mode shown in FIG. 4 . By installing the imaging device 10 in such a position, the imaging device 10 is installed at a higher position (movable body 401) than the cockpit 404. This allows for a wider range of images of the iron scrap to be captured, preventing the situation where only a portion of the scrap is captured. As a result, the presence or absence of foreign objects can be determined more accurately. Furthermore, the configurations shown in FIGS. 4 and 5 allow images to be captured from an angle different from that of the cockpit 404. Therefore, the operator can visually check from the cockpit 404 and determine the presence or absence of foreign objects based on multiple images captured from multiple angles other than the visual line of sight, thereby reducing the number of foreign objects that are missed. In the case of FIGS. 4 and 5, the imaging device 10 may be installed facing the tip of the movable body 401, or may be installed so as to face the area (area A) where the iron scrap to be determined is piled up when the heavy equipment 40 lifts the iron scrap, or may be installed so as to face the area (area B) where the iron scrap to be determined is piled up when the heavy equipment 40 lowers the iron scrap. The imaging device 10 may be equipped with a movable body for changing the imaging direction. When the heavy equipment 40 moves the iron scrap from area A to area B, the imaging device 10 may move the imaging direction to follow the movement of the iron scrap. For example, the imaging device 10 may operate to always follow the position of the tip portion of the movable body 401 (e.g., the lift magnet 402) and capture an image.Such a tracking operation of the imaging device 10 may be performed automatically by a device (not shown) or may be performed by a person (for example, an operator).

[0032] The output device 30 is used, for example, in the cockpit 404. The output device 30 may output information to an operator who operates the heavy equipment 40 in the cockpit 404. The output device 30 may be fixedly installed in the cockpit 404, or may be installed in a removable manner. If the output device 30 is removable, the output device 30 may be brought in from outside the cockpit 404 when the operator operates the heavy equipment 40. The output device 30 does not necessarily have to be installed in the cockpit 404 of the heavy equipment 40, and may be carried by a person outside the heavy equipment 40 (a worker other than the operator of the heavy equipment 40), for example.

[0033] 6 is a schematic block diagram showing a specific example of the functional configuration of the foreign object determination device 20. The input / output unit 21 acquires image data of the determination target image from the imaging device 10. The input / output unit 21 outputs determination result data to the output device 30. The input / output unit 21 may be configured using an interface with a bus or cable, or may be configured using a communication device. If the input / output unit 21 is a communication device, the input / output unit 21 may communicate data with another device via a network in accordance with the control of the control unit 23. In this case, the communication device may be a device that performs wireless communication or a device that performs wired communication.

[0034] The storage unit 22 stores data used by the control unit 23. The storage unit 22 may function as, for example, a foreign substance determination model storage unit 221. The foreign substance determination model storage unit 221 stores a foreign substance determination model. The foreign substance determination model is a function or logic accompanied by predetermined input / output information, which is used to determine whether or not an image of a predetermined foreign substance is present in an image to be determined.

[0035] The foreign matter determination model is generated in advance by a model construction process. Such a model construction process may be performed, for example, by another device, by the device itself (foreign matter determination device 20), or by human labor. The foreign matter determination model stored in the foreign matter determination model storage unit 221 will now be described.

[0036] The impurity determination model may be obtained, for example, by a model construction process using known data. The impurity determination model may be a trained model obtained by performing supervised learning using a plurality of known data (teacher data) having correct labels, a model obtained by performing statistical processing using known data, or other models. Specific examples of learning processes include multivariate analysis, so-called machine learning, deep learning, and the like. As the learning process, a learning process for classification or a learning process for regression may be used depending on the correct labels used in the teacher data. The control unit 23 determines the presence or absence of impurities in the steel scrap to be determined by using the impurity determination model stored in the impurity determination model storage unit 221. The teacher data in this embodiment will now be described.

[0037] The training data may be a combination of an image of the scrap iron being lifted by an attachment attached to the tip of the movable body of the heavy equipment and a correct answer label (information indicating the presence or absence of foreign matter). The images used as training data may be, for example, images captured under conditions similar to those of the imaging device 10 of the heavy equipment 40 to which the determination system 100 is applied. For example, if the image of the scrap iron captured by the imaging device 10 is the scrap iron being piled in area A or area B, the image of the piled scrap iron may be used as training data. For example, if the image of the scrap iron being lifted by the imaging device 10 is the scrap iron being lifted by a lift magnet, the image of the scrap iron being lifted by the lift magnet may be used as training data. In this case, the lifting means does not need to be limited to a lift magnet, and images of the scrap iron being lifted by other attachments may also be used as training data.

[0038] As the training data, images of the iron scrap containing foreign matter and images of the iron scrap not containing foreign matter may be used. The image of the iron scrap containing foreign matter may be an image captured when the iron scrap actually contains foreign matter, or an image captured when the iron scrap does not actually contain foreign matter and an image of the foreign matter superimposed by image processing may be used.

[0039] 7A and 7B are diagrams showing specific examples of images of iron scrap used as training data. Both Fig. 7A and Fig. 7B are images of iron scrap being lifted by a lift magnet 402. In Fig. 7A, the iron scrap does not contain any foreign matter. On the other hand, in Fig. 7B, the iron scrap contains the motor 81 as a foreign matter. In this way, an image captured without any foreign matter (e.g., Fig. 7A) and an image captured with a foreign matter (e.g., Fig. 7B) may be used as training data.

[0040] One type of foreign object determination model may be an image classification model that, when a single image such as that shown in FIGS. 7A and 7B is input, outputs whether the image contains a foreign object. If a foreign object is present, the model may also output the type of foreign object. Another type of foreign object determination model may, when a single image such as that shown in FIGS. 7A and 7B is input, output not only whether the image contains a foreign object but also, if a foreign object is present, information on where the foreign object is located in the image. Methods for outputting the location of such foreign objects include an object detection model that surrounds foreign objects in an image with a circumscribing rectangle and a segmentation model that fills in foreign objects in an image, and either of these may be used. The training data for the object detection model includes not only images such as those shown in FIGS. 7A and 7B, but also, for images in FIG. 7B that contain foreign objects, information on the circumscribing rectangle surrounding the foreign object. The training data for the segmentation model includes not only images such as those shown in Figures 7(A) and 7(B), but also, for Figure 7(B) which contains a foreign object, area information as a set of pixels corresponding to the position where the foreign object is located among all pixels in the image.

[0041] The type of foreign matter determination model is not limited to the model generated by the supervised learning described above. A known anomaly detection model may be used, which performs unsupervised learning on only images of steel scrap that do not contain foreign matter, generates a trained model that learns the characteristics of such steel scrap, and, in actual operation, when an image of steel scrap that contains foreign matter is input, detects parts containing foreign matter based on the absence of the trained characteristics. As a detection model or anomaly detection model based on such known supervised learning, for example, the model described in International Publication No. WO 2022 / 260133 may be used.

[0042] The processor of the control unit 23 executes the program to function as an information control unit 231 and a foreign object determination unit 232 .

[0043] The information control unit 231 controls the input and output of information. For example, the information control unit 231 acquires data (image data) of the determination target image from the imaging device 10 via the input / output unit 21. For example, the information control unit 231 outputs data (determination result data) indicating the determination result by the control unit 23 to the output device 30 via the input / output unit 21.

[0044] The foreign matter determination unit 232 performs foreign matter determination processing by acquiring the foreign matter determination model stored in the foreign matter determination model storage unit 221 and the determination target image acquired from the imaging device 10. The foreign matter determination processing determines whether or not the determination target image captured by the imaging device 10 contains an image of a foreign matter. In other words, it determines whether or not the iron scrap to be determined contains a predetermined foreign matter.

[0045] 8 is a flowchart showing a specific example of processing by the foreign matter determination device 20. First, the imaging device 10 captures an image (step S101), the information control unit 231 outputs image data from the imaging device 10 via the input / output unit 21, and the foreign matter determination unit acquires the image data (step S102). The foreign matter determination unit 232 performs a determination process using at least the foreign matter determination model and the image data (step S103). The information control unit 231 transmits information indicating the determination result to the output device 30 (step S104). Note that this transmission may be performed only when it is determined that a foreign matter is present, for example.

[0046] 9 is a diagram showing a first specific example of implementation of the determination system 100 of the first embodiment. In the determination system 100 shown in FIG. 9, the imaging device 10, the foreign object determination device 20, and the output device 30 are configured as a single determination device 200. The determination device 200 configured in this manner may further include an operation unit (such as a touch panel, buttons, or keyboard) that accepts operations from a user such as a worker. The user may be, for example, an operator of heavy machinery 40.

[0047] The determination device 200 may be operated by a user, and the imaging device 10 may operate in response to the user's operation to capture images of the iron scrap to be determined. Alternatively, the determination device 200 may repeatedly capture images or capture moving images without user operation. The determination target image generated by the imaging device 10 is input to the foreign matter determination device 20, which determines whether or not a foreign matter is present. The determination result is output to the user by the output device 30. Such a determination device 200 may be configured using, for example, a smartphone or tablet. In this case, the smartphone or tablet functions as the foreign matter determination device 20 by launching an application installed on the smartphone or tablet. Furthermore, the imaging device 10 and the output device 30 may be configured using devices that are already included in the smartphone or tablet, or may be configured using devices connected to the smartphone or tablet.

[0048] FIG. 10 is a diagram showing a second specific example of implementation of the determination system 100 of the first embodiment. In the determination system 100 shown in FIG. 10, the imaging device 10 and the foreign object determination device 20 are configured as the determination device 200, and the output device 30 is configured as another device. The determination device 200 configured in this manner may be provided in the same location as the imaging device 10 of the heavy equipment 40 in FIGS. 3 to 5. In this case, the determination device 200 and the output device 30 are connected so as to be able to communicate (for example, wirelessly). Therefore, the determination device 200 and the output device 30 may each further include a communication device.

[0049] The determination device 200 may be configured using, for example, a smartphone or a tablet, or may be configured as an imaging device with an information processing function added, such as a smart camera. Such information processing function may be built in the same housing as the imaging device 10, or may be built by connecting a single-board computer to the imaging device 10. The information processing function functions as the foreign object determination device 20. The output device 30 may also be configured using a smartphone or a tablet. In this case, the output device 30 may be configured using a device that is already included in the smartphone or tablet, or may be configured using a device connected to the smartphone or tablet.

[0050] FIG. 11 is a diagram showing a third specific example of implementation of the determination system 100 of the first embodiment. In the determination system 100 shown in FIG. 11, the foreign object determination device 20 and the output device 30 are configured as the determination device 200, and the imaging device 10 is configured as another device. The imaging device 10 configured in this manner may be installed, for example, in the same location as the imaging device 10 of the heavy equipment 40 in FIGS. 3 to 5. In this case, the imaging device 10 and the determination device 200 are connected so as to be able to communicate (for example, wirelessly). Therefore, the imaging device 10 and the determination device 200 may each further include a communication device.

[0051] The determination device 200 may be provided, for example, in a location where the output device 30 of the heavy equipment 40 in Figures 3 to 5 is installed. The determination device 200 receives a determination target image captured by the imaging device 10 from the imaging device 10 via communication and performs a determination process. The determination device 200 outputs the determination result by the output device 30. The determination device 200 may be configured using, for example, a smartphone or a tablet. In this case, the output device 30 may be configured using a device that the smartphone or tablet is already equipped with, or may be configured using a device connected to the smartphone or tablet.

[0052] Fig. 12 is a diagram showing a fourth specific example of implementation of the determination system 100 of the first embodiment. In the determination system 100 shown in Fig. 12, the imaging device 10 and the output device 30 are configured as a terminal device 300, and the foreign matter determination device 20 is configured as another device. The foreign matter determination device 20 and the terminal device 300 are communicably connected via a communication path such as a network 70. The foreign matter determination device 20 may be configured using an information processing device such as a server device or a cloud.

[0053] The terminal device 300 configured in this manner may be provided in a location where it is installed as the output device 30 of the heavy machinery 40 in FIGS. 3 to 5 . The terminal device 300 configured in this manner may further include an operation unit (such as a touch panel, buttons, or keyboard) that accepts operations from a user such as an operator. The user may be, for example, the operator of the heavy machinery 40. The terminal device 300 may be operated by a user, and the imaging device 10 may operate in response to the operation to capture images of the iron scrap to be determined. Furthermore, the terminal device 300 may repeatedly capture images or capture moving images independently of user operation. The determination target image generated by imaging with the imaging device 10 is transmitted to the foreign object determination device 20 via communication by the communication device.

[0054] When the foreign matter determination device 20 receives the determination target image, it determines whether or not the received determination target image contains a foreign matter. The foreign matter determination device 20 transmits information indicating the determination result to the terminal device 300. When the terminal device 300 receives the determination result from the foreign matter determination device 20, the output device 30 outputs the determination result to the user. Such a terminal device 300 may be configured using, for example, a smartphone or a tablet. Furthermore, the imaging device 10 and the output device 30 may each be configured using devices that are already included in the smartphone or tablet, or may be configured using devices connected to the smartphone or tablet.

[0055] 9 to 12 have been used to describe specific examples of implementation of the determination system 100 of the first embodiment, but the implementation of the determination system 100 does not need to be limited to the specific examples described above. For example, as shown in FIG. 1, the image capture device 10, foreign matter determination device 20, and output device 30 may be implemented as different devices. In this case, for example, the image capture device 10 and the output device 30 may be installed as shown in FIGS. 3 to 5, and the foreign matter determination device 20 may be configured using an information processing device 90 such as a server device or a cloud. In this case, the image capture device 10, foreign matter determination device 20, and output device 30 may each further include a communication device.

[0056] An example of the operation of the determination system 100 configured as described above will be described. For example, the imaging device 10 may be configured to repeatedly capture images at a predetermined timing and transmit the images to be determined obtained by the imaging to the foreign matter determination device 20 according to a certain rule (sequentially or at regular intervals). In this case, the operator of the heavy equipment 40 moves the iron scrap from area A to area B without being particularly aware of the imaging. When the output device 30 outputs information indicating that a foreign matter is contained, the operator of the heavy equipment 40 may remove the foreign matter from the iron scrap to be determined.

[0057] The operator of the heavy equipment 40 approaches area A by maneuvering the heavy equipment 40 to move it. The operator of the heavy equipment 40 approaches area A and maneuvers the heavy equipment 40 so that the imaging device 10 is facing area A, and then operates the imaging device 10. Such operation of the imaging device 10 may be performed, for example, by directly touching a button or touch panel provided on the imaging device 10, or by using an operating device such as a remote controller connected to the imaging device 10 via short-range wireless communication, or by the operator uttering a specific phrase (e.g., "take a picture") using voice recognition. By performing such an operation, an image of the iron scrap piled in area A may be captured. The imaging device 10 may capture images of the iron scrap in area A without any particular operation by the operator, for example, by repeatedly capturing images or capturing a moving image.

[0058] The operator of the heavy equipment 40 operates the heavy equipment 40 (particularly the movable body 401) to lift the iron scrap with the heavy equipment 40. When operating the imaging device 10, the operator of the heavy equipment 40 may capture an image of the iron scrap being lifted by operating the imaging device 10 while lifting the iron scrap in this manner. FIG. 13 is a diagram showing a specific example of the state of the heavy equipment 40. In FIG. 13, the imaging device 10 captures an image with the lens facing diagonally upward or directly upward so that the lift magnet 402 is included in the imaging range when the movable body 401 is raised. The imaging device 10 may be installed in a fixed position in this orientation, or may be installed as a device with a movable body so that the lens orientation can be changed. By capturing an image with the imaging device 10 in this state, it is possible to capture an image of the lifted iron scrap as viewed from below. In particular, when the lift magnet 402 is used, iron scrap is attached below, making it possible to capture an image that captures a larger amount of iron scrap. Capturing such an image makes it possible to more accurately determine the presence or absence of foreign matter. Furthermore, if the imaging device 10 is installed facing the tip of the movable body 401, an image of the lifted iron scrap may be captured, for example, by repeatedly capturing images or capturing a moving image, without any particular operation by the operator. When the imaging device 10 captures an image in the state shown in FIG. 13 , the image is captured while the iron scrap is being lifted. This increases the likelihood that the iron scrap will fall from the tip of the movable body 401 and collide with the lens of the imaging device 10, for example. Furthermore, it takes time to bring the movable body 401 of the heavy equipment into the state shown in FIG. 13 , which may result in reduced productivity. Despite the possibility of such a collision or reduced productivity, this configuration may be adopted in this embodiment because it enables more accurate determination of the presence or absence of foreign matter, as described above.

[0059] The operator of the heavy equipment 40 operates the heavy equipment 40 (particularly the movable body 401) to lift the iron scrap with the heavy equipment 40. The operator of the heavy equipment 40 may operate the heavy equipment 40 (particularly the movable body 401) so that a wider area of ​​the lifted iron scrap is captured by the imaging device 10. For example, the operator may operate the heavy equipment 40 so that the iron scrap holding portion (e.g., the tip portion of the movable body 401) of the heavy equipment 40 lifting the iron scrap is directed toward the imaging device 10. A more specific example of such operation is operation in which the bottom surface (the surface on which the iron scrap is attached) of the tip of the movable body 401 (e.g., the lift magnet 402) is directed toward the imaging device 10. FIG. 14 is a diagram showing a specific example of the state of the heavy equipment 40. By capturing an image with the imaging device 10 in this state, it is possible to capture an image that captures a larger portion of the lifted iron scrap. In FIG. 14, the imaging device 10 captures an image with the lens facing horizontally, diagonally downward, or diagonally upward so that the lift magnet 402 is included in the imaging range. The imaging device 10 may be installed fixedly in this orientation, or may be installed as a device with a movable body so that the orientation of the lens can be changed. Capturing such images enables more accurate determination of the presence or absence of foreign objects. When the imaging device 10 and the operator are facing the same direction, the operator can confirm the presence or absence of foreign objects with their own eyes by operating the imaging device 10 in this manner, thereby performing a double check using the determination system and visual inspection. When the operator operates the imaging device 10, the operator may operate the imaging device 10 while operating it in this manner to capture an image in which the bottom surface of the lift magnet 402 is facing the imaging device 10. Furthermore, when the imaging device 10 is installed facing the tip of the movable body 401, images of the iron scrap in the above-described state may be captured, for example, by repeatedly capturing images or capturing moving images, without any particular operation by the operator.

[0060] The operator of the heavy equipment 40 may operate the heavy equipment 40 (particularly the movable body 401) to lower the scrap iron once lifted from area A into an area (area C) different from areas A and B. FIG. 15 is a diagram showing an outline of such area C. In FIG. 15, the imaging device 10 captures an image with the lens facing horizontally or diagonally downward so as to include area C in the imaging range. The imaging device 10 may be installed in a fixed orientation, or may be installed as a device with a movable body so that the orientation of the lens can be changed. When lowering the scrap iron into area C, the operator of the heavy equipment 40 may operate the heavy equipment (particularly the movable body 401) to scatter the scrap iron components over a wide area so as to minimize the accumulation of individual scrap iron components. By operating in this manner, it is possible to capture an image to be determined in which the individual scrap iron components are more clearly visible. For example, as in the embodiments shown in FIGS. 13 and 14 , when capturing an image of a lump of iron scrap held at the tip of the movable body 401, it is difficult to detect foreign objects hidden inside the lump. In this embodiment, to address this issue, the iron scrap members are first scattered in area C, eliminating such missed detections. Therefore, the presence or absence of foreign objects can be determined more accurately. Furthermore, when the imaging device 10 and the operator are facing the same direction, this operation allows the operator to visually confirm the presence or absence of foreign objects, thereby enabling a double check between the detection system and visual inspection. When the operator operates the imaging device 10, the operator may operate the imaging device 10 while operating the heavy equipment 40 in this manner to capture images of the individual iron scrap members scattered around. Furthermore, images of the iron scrap in the above state may be captured without any specific operator operation, for example, by repeatedly capturing images or capturing moving images. In the case where the iron scrap is temporarily lowered into area C and photographed, after photographing in area C, the operator collects the iron scrap placed in area C and moves it to area B.

[0061] The following describes the features of the determination system 100 of the first embodiment configured as described above. Conventionally, as a system in which the movement path of a conveying device is fixed (hereinafter referred to as a "fixed-path system"), capturing an image of the iron scrap somewhere along the path was assumed, and therefore, the imaging device was installed in a fixed location. Therefore, in a system in which the conveying path of the iron scrap is not fixed, such as when the iron scrap is conveyed by heavy equipment 40, it is difficult to accurately determine the presence or absence of foreign matter. In contrast, the determination system 100 of the first embodiment acquires images of the iron scrap placed in the source area A or the destination area B, or the iron scrap during the conveying process by the heavy equipment 40, using an imaging device installed in the heavy equipment 40 to obtain images, and determines whether the iron scrap contains foreign matter. The conveying process refers to, for example, a process in which the iron scrap is gripped in a certain area (e.g., area A) by a conveying tool attached to the heavy equipment 40, transported from that area to another area (e.g., area B or area C), and then released from the grip in the destination area. Because the imaging device 10 moves with the movement of the heavy equipment 40, it can always capture images closer to the iron scrap than if the imaging device 10 were installed in a fixed position, making it possible to accurately determine whether or not foreign matter is mixed in the iron scrap. Furthermore, because the imaging device 10 moves with the movement of the heavy equipment 40, it is possible to capture images while always maintaining approximately the same distance between the imaging device 10 and the iron scrap. This type of imaging provides the following advantages. For example, when determining whether or not foreign matter is present using a trained model obtained by a learning process using teacher images, it is possible to capture images from approximately the same distance and at approximately the same size as the foreign matter captured in the image used as the teacher image. Therefore, it is possible to accurately determine whether or not foreign matter is mixed in the iron scrap to be recycled. This advantage is not limited to when a trained model is used, but can also be achieved when using techniques such as pattern matching. Furthermore, because the distance is always approximately the same, setting the camera focus to match the iron scrap allows for consistently well-focused images.Furthermore, the operator's cabin 404 of the heavy equipment 40 is often located relatively closer to the scrap iron than the operator's cabin of a fixed-route system, such as an overhead crane. Furthermore, in fixed-route systems, the operator's cabin is often located at an elevated position, so the operator must look down from above to visually inspect the scrap iron. In such cases, the equipment holding the scrap iron is often in the operator's field of vision, making it difficult to visually inspect the scrap iron. Therefore, transporting the scrap iron using the heavy equipment 40 instead of a fixed-route system has the advantage of allowing the operator to visually inspect the scrap iron from a closer position. Furthermore, the introduction cost of the heavy equipment is often relatively low and short compared to the introduction cost of a fixed-route system. Therefore, few companies that handle scrap iron generally have fixed-route systems. Therefore, the technology for handling scrap iron using heavy equipment, as in the present embodiment, also has the advantage of enabling more companies to improve the accuracy of detecting foreign objects mixed in the scrap iron.

[0062] Second Embodiment FIG. 16 is a schematic block diagram showing the system configuration of a determination system 100 according to a second embodiment of the present invention. The determination system 100 according to the second embodiment differs from the determination system 100 according to the first embodiment in that it further includes a timing determination device 50. The timing determination device 50 determines the timing at which a determination target image used by the foreign object determination device 20 for determination processing is acquired. Therefore, while the first embodiment is an embodiment in which the user determines the timing at which the determination target image is acquired, or in which the determination target image is acquired according to a certain rule (for example, a certain interval), the second embodiment is an embodiment in which the timing at which the determination target image is acquired is variable (free), and is determined by the determination system 100. Details of the timing determination device 50 will be described below.

[0063] FIG. 17 is a schematic block diagram showing a specific example of the functional configuration of the timing determination device 50. The input / output unit 51 acquires image data of the determination target image from the imaging device 10. The input / output unit 51 inputs data used for timing determination processing from another device. For example, the input / output unit 51 inputs information regarding the control of the heavy equipment 40 (hereinafter referred to as "control information") from the heavy equipment 40 on which the imaging device 10 is installed. The control information may be, for example, information indicating the operation details performed by the operator, electrical signals or control signals of electrical equipment installed in the heavy equipment 40, or hydraulic signals of hydraulic equipment installed in the heavy equipment 40. The input / output unit 51 outputs information indicating the timing determination result to another device (e.g., the imaging device 10 or the foreign object determination device 20). The input / output unit 51 may output image data at the timing indicated by the determination result to the foreign object determination device 20. The input / output unit 51 may be configured using an interface with a bus or cable, or may be configured using a communication device. If the input / output unit 51 is a communication device, the input / output unit 51 may communicate data with another device via a network under the control of the control unit 53. In this case, the communication device may be a device that performs wireless communication or a device that performs wired communication.

[0064] The storage unit 52 stores data used by the control unit 53. The storage unit 52 may function, for example, as a timing judgment model storage unit 521. The timing judgment model storage unit 521 stores a timing judgment model. The timing judgment model is a function or logic accompanied by predetermined input / output information, which is used to determine the timing at which the judgment target image used by the foreign object judgment device 20 for the judgment process is acquired.

[0065] The timing judgment model is generated in advance by a model construction process, which may be performed by another device, the timing judgment device 50 itself, or a human.

[0066] The processor of the control unit 53 executes a program to function as an information control unit 531 and a timing determination unit 532 .

[0067] The information control unit 531 controls the input and output of information. For example, the information control unit 531 acquires data (image data) of the image to be determined from the imaging device 10 via the input / output unit 51. For example, the information control unit 531 acquires control information from the heavy equipment 40. For example, the information control unit 531 outputs data indicating the determination result by the control unit 53 to the imaging device 10 or the foreign object determination device 20 via the input / output unit 51.

[0068] The timing determination unit 532 performs timing determination processing using the timing determination model stored in the timing determination model storage unit 521, and the determination target image acquired from the imaging device 10 and the control information acquired from the heavy machinery 40. By performing such timing determination processing, the timing determination unit 532 performs determination processing in real time on the image captured by the imaging device 10 and the control information acquired from the heavy machinery 40, and outputs the determination results as appropriate via the input / output unit 51. Below, the timing determination models stored in the timing determination model storage unit 521 will be described.

[0069] The timing determination model may input one or more pieces of control information for the heavy machine 40 and determine whether the timing is the time when the determination target image used in the determination process by the foreign object determination device 20 was acquired, based on whether the control information satisfies a predetermined condition. The predetermined condition may be, for example, whether one piece of control information matches a certain value, is equal to or greater than (equal to) a certain value, exceeds (below) a certain value, or falls within a certain range. Multiple such conditions may be set for multiple pieces of control information, and the condition may be set using a logical expression that combines these conditions.

[0070] Several specific examples of such control information will be described. As a first example, there is control information indicating that the movable body 401 has been operated to lift iron scrap, as shown in FIG. 13 . Such control information may be information indicating the operation performed by the operator of the heavy equipment 40 (e.g., an electrical signal indicating the operation of a control lever). For example, the time period during which operation information (electrical signal) indicating that the operator of the heavy equipment 40 is operating the movable body 401 to lift iron scrap may be determined to be the timing at which the determination target image is acquired. Such control information may be a hydraulic signal in the heavy equipment 40. The time period during which a hydraulic signal is obtained to hydraulically move the movable body 401 so that the tip of the movable body 401 moves upward, and the operation corresponding to the hydraulic signal is completed, and a new hydraulic signal is obtained to operate the movable body 401, may be determined to be the timing at which the determination target image is acquired. Furthermore, a plurality of such electrical signals and hydraulic signals may be combined to determine the timing at which the determination target image to be used in the determination process is acquired.

[0071] As a second example, as shown in FIG. 14 , there is control information indicating that the bottom surface (surface on which iron scrap is attached) of the tip of the movable body 401 (e.g., the lift magnet 402) has been operated toward the imaging device 10. Such control information may be information indicating the operation by the operator of the heavy equipment 40 (e.g., an electrical signal indicating the operation of a control lever). For example, the time period during which operation information (electrical signal) indicating that the operator of the heavy equipment 40 is operating the movable body 401 to operate the tip of the movable body 401 toward the imaging device 10 is obtained may be determined to be the timing at which the determination target image is acquired. Such control information may be a hydraulic signal in the heavy equipment 40. The time period during which a hydraulic signal for hydraulically moving the movable body 401 so that the tip of the movable body 401 is operated toward the imaging device 10 is obtained, and the operation corresponding to the hydraulic signal is completed, and a new hydraulic signal for operating the movable body 401 is obtained may be determined to be the timing at which the determination target image is acquired. Furthermore, a plurality of such electrical signals and hydraulic signals may be combined to determine the timing at which the determination target image to be used in the determination process is acquired.

[0072] As a third example, another specific example of control information is control information indicating that an operation was performed to scatter the individual scrap iron components into area C so that they do not pile up, as shown in FIG. 15 . For example, if a rule requires that scrap iron be lowered from area A to area C, such control information may indicate that the scrap iron was first lifted and then lowered. More specifically, the control information may be as follows. Such control information may be information indicating the operation performed by the operator of the heavy equipment 40 (e.g., an electrical signal indicating an operation of the control lever). For example, if operation information (electrical signal) is obtained indicating that the operator of the heavy equipment 40 performed an operation to hold the scrap iron at the tip of the movable body 401 (e.g., an operation to generate magnetic force with the lift magnet 402) and then performed an operation to release the holding of the scrap iron (e.g., an operation to stop the generation of magnetic force with the lift magnet 402), it may be determined that the time period until the next operation to start holding the scrap iron is the timing for acquiring the target image. Such control information may also be a hydraulic signal in the heavy equipment 40. When a hydraulic signal for gripping the iron scrap (e.g., a hydraulic signal for closing the attachment for gripping the iron scrap) is obtained at the tip of the movable body 401, and then a hydraulic signal for releasing the grip of the iron scrap (e.g., a hydraulic signal for opening the attachment) is obtained, the time period until the next hydraulic signal for starting to grip the iron scrap is obtained may be determined to be the timing when the determination target image to be used in the determination process is acquired. Furthermore, a plurality of the above-mentioned electric signals and hydraulic signals may be combined to determine the timing when the determination target image to be used in the determination process is acquired.

[0073] The timing judgment model may input, for example, an image captured by the imaging device 10 and output a judgment result indicating whether or not a predetermined timing has occurred. One example of such a timing judgment model is a model that determines whether an image captured by the imaging device 10 is an image captured from below while the heavy equipment 40 is lifting the iron scrap. Such a model may be obtained as a trained model by performing a learning process using, for example, images captured in such a state as training data. FIG. 18(A) is a diagram showing a specific example of training data used for such a learning process. FIG. 18(A) shows an image captured from below while the iron scrap is being held by the lift magnet 402 and the movable body 401 connected to the lift magnet 402 is moving upward. By performing a learning process using, as training data, an image of the iron scrap being actually lifted by the heavy equipment 40 as viewed from below, a trained model that can determine such a state from an image can be obtained. Such a trained model may be used as the timing judgment model. More simply, it is possible to pre-calculate feature quantities showing a predetermined pattern for an image taken while looking up at the iron scrap from below while it is being lifted by heavy equipment 40, and then calculate the same feature quantities for a new image that is newly taken by imaging device 10 and input to timing determination unit 532, and compare the calculated feature quantities with the image feature quantities calculated in advance to determine whether the new image was taken while looking up at the iron scrap from below while it is being lifted by heavy equipment 40.

[0074] Another specific example of a timing judgment model is a model that determines whether an image captured by the imaging device 10 was captured with the bottom surface (the surface on which the scrap iron is attached) of an attachment (e.g., the lift magnet 402) attached to the tip of the movable body 401 facing the imaging device 10. Such a model may be obtained as a trained model by performing a learning process using, for example, images captured in such a state as training data. FIG. 18(B) is a diagram showing a specific example of training data used for performing such a learning process. FIG. 18(B) shows an image captured with the lift magnet 402 holding scrap iron and with the bottom surface of the lift magnet 402 facing the imaging device 10. By performing a learning process using, as training data, images of the lift magnet 402 actually facing the bottom surface of the lift magnet 402 facing the imaging device 10, a trained model that can determine such a state from an image can be obtained. Such a trained model may be used as a timing judgment model. More simply, it is possible to calculate in advance feature quantities that indicate a predetermined pattern for an image captured with the bottom surface of the attachment facing the imaging device 10, and then calculate the same feature quantities for a new image that is captured by the imaging device 10 and input to the timing determination unit 532, and compare them with the image feature quantities that were calculated in advance to determine whether the new image was captured with the bottom surface of the attachment facing the imaging device 10.

[0075] Another specific example of a timing judgment model is a model that determines whether an image captured by the imaging device 10 shows scrap iron scattered in area C without stacking. In other words, this timing judgment model determines whether an image was captured of scrap iron lifted at least once by the heavy equipment 40 and then placed in an area (e.g., area C) different from the destination (e.g., area B) before being transported to the destination. Such a model may be obtained as a trained model by performing a learning process using images captured in such a state as training data. FIG. 18(C) shows a specific example of training data used for such a learning process. In FIG. 18(C), scrap iron is scattered on the ground, and the image is obtained by capturing the state. By performing a learning process using images of actual scrap iron scattered on the ground as training data, a trained model that can determine such a state from an image can be obtained. Such a trained model may be used as a timing judgment model. More simply, feature quantities showing a predetermined pattern may be calculated in advance for an image of iron scrap scattered on the ground, and the same feature quantities may be calculated for a new image newly captured by the imaging device 10 and input to the timing determination unit 532, and the new image may be compared with the previously calculated image feature quantities to determine whether or not the new image is an image of iron scrap scattered on the ground. In this way, in the above description, a method has been described in which the timing for determination is determined mechanically using a timing determination model from electrical signals, hydraulic signals, or situation identification of the captured image.

[0076] 19 is a flowchart showing a specific example of processing by the determination system 100 in the second embodiment. The imaging device 10 continuously captures images (step S201). The timing determination device 50 determines the image acquisition timing (hereinafter also referred to as "processing timing") at which a determination target image to be used in the determination process is acquired (step S202). The foreign substance determination device 20 waits until it is determined that it is the image acquisition timing. If it is determined that it is the image acquisition timing (step S202-YES), the foreign substance determination device 20 acquires image data captured at that timing or immediately before (step S203). Then, it performs determination processing using the acquired image data and the foreign substance determination model (step S204). Then, the foreign substance determination device 20 transmits information indicating the determination result to the output device 30 (step S205).

[0077] Next, a specific example of implementation of the determination device 200 will be described using Figures 20 to 24. Note that the determination device 200, image acquisition device 400, terminal device 300 shown in Figures 20 to 24, and devices connected to these devices (imaging device 10, output device 30, image acquisition device 400), and terminal device 300 may each be configured using, for example, a smartphone or a tablet. In this case, an application installed on the smartphone or tablet is started, causing the smartphone or tablet to function as the determination device 200, etc. Furthermore, devices connected to the determination device 200, etc. may be configured using equipment connected to the smartphone or tablet.

[0078] Fig. 20 is a diagram showing a first specific example of implementation of the determination system 100 of the second embodiment. In the determination system 100 shown in Fig. 20, the imaging device 10, the foreign object determination device 20, the output device 30, and the timing determination device 50 are configured as a single determination device 200. The determination device 200 configured in this manner may further include an operation unit (such as a touch panel, buttons, or keyboard) that accepts operations from a user such as a worker. The user may be, for example, an operator of heavy machinery 40.

[0079] The imaging device 10 of the determination device 200 captures images at a predetermined timing without user operation. The imaging device 10 may continuously capture moving images. The timing determination device 50 determines the timing at which a determination target image is acquired based on input data (images captured by the imaging device 10 and control information for the heavy equipment 40). When the foreign object determination device 20 determines that it is time to acquire a determination target image to be used in the determination process, the timing determination device 50 outputs the image at that timing (determination target image) to the foreign object determination device 20. The foreign object determination device 20 determines the presence or absence of a foreign object from the image captured at that timing (determination target image). The determination result is output to the user by the output device 30.

[0080] In the first specific example, the determination device 200 may be operated by a user, and the imaging device 10 may operate in response to the user's operation to capture an image of the iron scrap to be determined. In this case, the foreign matter determination device 20 also performs processing in accordance with the determination result of the timing determination device 50.

[0081] Fig. 21 is a diagram showing a second specific example of implementation of the determination system 100 of the second embodiment. In the determination system 100 shown in Fig. 21, the imaging device 10, the foreign object determination device 20, and the timing determination device 50 are configured as a determination device 200, and the output device 30 is configured as another device. In this case, the determination device 200 configured in this manner is connected to the output device 30 so that they can communicate (for example, wirelessly). Therefore, the determination device 200 and the output device 30 may each further include a communication device.

[0082] The determination device 200 may be configured using, for example, a smartphone or tablet, or may be configured as an imaging device with an information processing function added, such as a smart camera. Such information processing function may be built in the same housing as the imaging device 10, or may be built by connecting a single-board computer to the imaging device 10. The information processing function functions as the foreign object determination device 20 and the timing determination device 50.

[0083] The timing determination device 50 determines the timing at which a determination target image will be acquired based on input data (images captured by the imaging device 10 and control information for the heavy machinery 40). When it determines that it is time for the foreign object determination device 20 to acquire a determination target image to be used in the determination process, the timing determination device 50 outputs the image at that timing (determination target image) to the foreign object determination device 20. The foreign object determination device 20 acquires the image captured at that timing (determination target image) and determines whether or not a foreign object is present. The determination result is output to the user by the output device 30.

[0084] FIG. 22 is a diagram showing a third specific example of implementation of the determination system 100 of the second embodiment. In the determination system 100 shown in FIG. 22, the imaging device 10 and timing determination device 50 are configured as an image acquisition device 400, and the foreign object determination device 20 and output device 30 are configured as a determination device 200. The image acquisition device 400 configured in this manner may be installed, for example, in the same location as the imaging device 10 of the heavy equipment 40 in FIGS. 3 to 5. In this case, the image acquisition device 400 and the determination device 200 are connected so as to be able to communicate (for example, wirelessly). Therefore, the image acquisition device 400 and the determination device 200 may each further include a communication device.

[0085] The image acquisition device 400 transmits the image captured by the imaging device 10 to the determination device 200 at the timing determined by the timing determination device 50. The determination device 200 may be provided, for example, in a location where it is installed as the output device 30 of the heavy machinery 40 in FIGS. 3 to 5 . The determination device 200 receives the determination target image from the image acquisition device 400 via communication and performs a determination process. The determination device 200 outputs the determination result via the output device 30.

[0086] FIG. 23 is a diagram showing a fourth specific example of implementation of the determination system 100 of the second embodiment. In the determination system 100 shown in FIG. 23, the imaging device 10 and timing determination device 50 are configured as an image acquisition device 400, and a foreign object determination device 20 and an output device 30 are provided, respectively. The image acquisition device 400 configured in this manner may be provided, for example, in the same location as the imaging device 10 of the heavy equipment 40 in FIGS. 3 to 5. In this case, the image acquisition device 400 and the foreign object determination device 20 are connected so as to be able to communicate (e.g., wirelessly). Therefore, the image acquisition device 400 and the foreign object determination device 20 may each further include a communication device. The foreign object determination device 20 may be configured using an information processing device 90, such as a server device or a cloud.

[0087] The image acquisition device 400 transmits the image captured by the imaging device 10 to the foreign object determination device 20 at the timing determined by the timing determination device 50. The foreign object determination device 20 may be provided, for example, in the location where it is installed as the output device 30 of the heavy equipment 40 in FIGS. 3 to 5 . The foreign object determination device 20 receives the determination target image from the image acquisition device 400 via communication and performs determination processing. The foreign object determination device 20 outputs the determination result via the output device 30.

[0088] Fig. 24 is a diagram showing a fifth specific example of implementation of the determination system 100 of the second embodiment. In the determination system 100 shown in Fig. 24, the imaging device 10, the timing determination device 50, and the output device 30 are configured as a terminal device 300, and the foreign matter determination device 20 is configured as another device. The foreign matter determination device 20 and the terminal device 300 are communicatively connected via a communication path such as a network. The foreign matter determination device 20 may be configured using an information processing device 90 such as a server device or a cloud.

[0089] The terminal device 300 configured in this manner may be installed in a location such as the location where it is installed as the output device 30 of the heavy equipment 40 in Figures 3 to 5. The imaging device 10 of the terminal device 300 captures images at a predetermined timing without user operation. The imaging device 10 may capture moving images continuously. The timing determination device 50 determines the timing based on input data (images captured by the imaging device 10 and control information for the heavy equipment 40). When it is determined that the timing has come for the foreign object determination device 20 to acquire a determination target image to be used in the determination process, the terminal device 300 transmits image data to the foreign object determination device 20.

[0090] When the foreign matter determination device 20 receives image data from the terminal device 300, it determines the presence or absence of a foreign matter using the image. The determination result is output to the user by the output device 30. Such a determination device 200 may be configured using, for example, a smartphone or a tablet.

[0091] Specific examples of implementation of the determination system 100 of the second embodiment have been described above using FIGS. 20 to 24 , but the implementation of the determination system 100 is not limited to the specific examples described above. For example, as shown in FIG. 16 , the image capture device 10, foreign object determination device 20, output device 30, and timing determination device 50 may each be implemented as separate devices. In this case, for example, the image capture device 10 and the output device 30 may be installed as shown in FIGS. 3 to 5 , and the foreign object determination device 20 may be configured using an information processing device 90 such as a server device or a cloud. The timing determination device 50 may be configured to be connected to the image capture device 10 without a network. In this case, the image capture device 10, foreign object determination device 20, output device 30, and timing determination device 50 may each further include a communication device.

[0092] In the determination system 100 of the second embodiment configured as described above, the timing determination device 50 determines the timing at which an appropriate image is likely to be captured. Then, images captured at the timing of the determination result are acquired and foreign object determination is performed. This enables accurate determination of the presence or absence of foreign objects in iron scrap to be recycled. Furthermore, only images captured at the timing of the determination result are transmitted to the foreign object determination device 20 via the network. This reduces the resources and communication data volume required for transmitting the images to be determined. Furthermore, by determining the timing and focusing processing on that timing, power consumption can be reduced. In particular, in environments where power capacity is limited, such as in heavy machinery, reducing power consumption has the advantage of enabling a compact device by operating the image capture device, foreign object determination device, and output device all via wiring and power supply from the heavy machinery, and of enabling the installation of multiple cameras as a result of reducing power consumption per camera. Furthermore, by determining the timing and focusing processing on that timing, it is possible to prevent false positives from being reported to workers. In other words, if a false detection occurs due to processing by the foreign matter determination device 20 at a time when iron scrap is clearly not being imaged, it is possible to prevent such a false detection from being output to the worker.

[0093] 25 is a diagram showing an outline of an example of the hardware configuration of an information processing device 90 applied to this embodiment. The information processing device 90 includes a processor 91, a main storage device 92, a communication interface 93, an auxiliary storage device 94, an input / output interface 95, and an internal bus 96. The processor 91, the main storage device 92, the communication interface 93, the auxiliary storage device 94, and the input / output interface 95 are connected to each other via the internal bus 96 so as to be able to communicate with each other.

[0094] Specific examples of such information processing devices 90 include smartphones, tablets, personal computers, server devices, programmable logic controllers (PLCs), and dedicated devices. The processor 91 and main memory device 92 may be configured using general-purpose devices such as central processing units (CPUs). They may also be configured using application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or field programmable gate arrays (FPGAs). The programs executed by the processor 91 may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and semiconductor storage devices (e.g., SSDs), as well as storage devices such as hard disks and semiconductor storage devices built into computer systems. The programs may be transmitted via telecommunications lines. The communication interface 93 is configured using a communication device. The auxiliary storage device 94 is configured using a storage device such as a magnetic hard disk drive or semiconductor storage device. The input / output interface 95 is an interface between external devices and the information processing device 90.

[0095] The information processing device 90 may be applied to, for example, the foreign object determination device 20 and the timing determination device 50. In this case, for example, the input / output unit 21 and the input / output unit 51 may be configured using a communication interface 93, or may be configured using an input / output interface 95. For example, the memory unit 22 and the memory unit 52 may be configured using an auxiliary memory device 94. Furthermore, the control unit 23 and the control unit 53 may be configured using a processor 91 and a main memory device 92. Furthermore, the information processing device 90 may be applied to the determination device 200 and the image acquisition device 400.

[0096] (Modification) FIG. 26 is a flowchart showing a first modification of the processing of the determination system 100 in the second embodiment. First, the timing determination device 50 determines the image capture timing (also referred to as processing timing) for capturing an image using one or more pieces of control information for the heavy equipment 40 (step S301). The other devices in the determination system 100 wait until it is determined that it is time to capture an image. If it is determined that it is time to capture an image (step S301—YES), the image capture device 10 captures an image at that timing (step S302). The foreign object determination device 20 acquires the captured image data (step S303). Then, it performs a determination process using the acquired image data and a foreign object determination model (step S304). Then, the foreign object determination device 20 transmits information indicating the determination result to the output device 30 (step S305). In this way, by operating all steps—image capture, image acquisition, foreign object determination process, and result output—only at the image capture timing, it is possible to reduce power consumption.

[0097] FIG. 27 is a flowchart illustrating a second modified example of the processing of the determination system 100 according to the second embodiment. The imaging device 10 continuously captures images (step S401). The foreign object determination device 20 continuously acquires image data captured by the imaging device 10 (step S402). Then, a determination process is performed using the acquired image data and the foreign object determination model (step S403). The timing determination device 50 determines the output timing (also referred to as processing timing) for outputting the determination result based on the foreign object determination model using one or more pieces of control information for the heavy equipment 40 or the captured image data (step S404). The foreign object determination device 20 waits to output the determination result until it is determined that the output timing has arrived. If it has arrived that the output timing has arrived (step S404—YES), the foreign object determination device 20 transmits information indicating the determination result at that time to the output device 30 (step S405). This operation prevents the operator of the heavy equipment 40 from receiving an erroneous determination result based on an image that clearly does not contain scrap iron.

[0098] As described above, in the second embodiment, the timing determination unit 532 of the timing determination device 50 may perform timing determination processing based on control information acquired from the heavy equipment 40, without using images captured by the imaging device 10. In this case, the timing determination unit 532 may control the imaging device 10 to perform imaging in the imaging device 10 at the determined timing. Control of the imaging device 10 by the timing determination unit 532 may be performed, for example, by a processor of the timing determination device 50 executing a program.

[0099] Alternatively, the foreign object determination device 20 may acquire the determination target image as follows: The image acquisition device 400 stores the images captured by the imaging device 10 in a storage device (for example, the storage unit 52 of the timing determination device 50 or a storage unit of another information processing device). The foreign object determination device 20 accesses the storage device and acquires the determination target image from the images stored in the storage device based on the timing determined by the timing determination device 50.

[0100] The "image acquisition timing," "image capture timing," and "output timing" described in the second embodiment above are collectively referred to as "processing timing."

[0101] As a modification from another perspective applicable to both the first and second embodiments, although the above description focuses on images of iron scrap transported by heavy equipment 40 as the target of assessment, iron scrap may be transported by other devices. For example, the assessment system 100 may be applied to a configuration in which iron scrap is transported by an overhead crane or other device. In this case, the imaging device 10 may be attached to, for example, a part of a moving crane. Specifically, the crane includes a crane body capable of traveling in both directions (traveling direction) in a predetermined direction, a trolley capable of moving (transversely) on the crane body in a direction perpendicular to the traveling direction (transverse direction) on a horizontal plane including the traveling direction, and a lifting tool attached to the trolley for lifting the transported object. However, at least one camera may be attached to the crane body, the trolley, or a holding tool for holding the transported object on the lifting tool. In other words, the imaging device 10 may be attached to a transporting machine, which may be a heavy equipment 40 or a crane.

[0102] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0103] DESCRIPTION OF SYMBOLS 100... Determination system, 10... Imaging device, 20... Foreign object determination device, 30... Output device, 40... Heavy machinery, 401... Movable body, 402... Lift magnet, 403... Main body, 404... Cockpit, 405... Rotating body, 406... Running body, 21... Input / output unit, 22... Memory unit, 221... Foreign object determination model memory unit, 23... Control unit, 231... Information control unit, 232... Foreign object determination unit, 200... Determination device, 300... Terminal device, 51... Input / output unit, 52... Memory unit, 521... Timing determination model memory unit, 53... Control unit, 531... Information control unit, 532... Timing determination unit, 400... Image acquisition device

Claims

1. A judgment system comprising: an imaging device provided on a conveying machine for conveying iron scrap; and a foreign matter judgment device that acquires an image of an object to be judged captured by the imaging device and judges whether the iron scrap contains a foreign matter.

2. The determination system according to claim 1, wherein the imaging device is provided on a main body or a movable body of the conveying machine.

3. A judgment system as described in claim 1 or 2, wherein the imaging device photographs the iron scrap during the process of being transported by the transport machine.

4. A judgment system as claimed in any one of claims 1 to 3, further comprising a timing judgment device which judges a processing timing which is any one of the following timings: an image acquisition timing for acquiring the judgment target image, an imaging timing for imaging the judgment target image, or an output timing for outputting a judgment result by the foreign matter judgment device, wherein when the image acquisition timing is judged by the timing judgment device, the foreign matter judgment device acquires an image captured at the image acquisition timing from among images captured by the imaging device, when the imaging timing is judged by the timing judgment device, imaging is performed by the imaging device at the imaging timing, and when the output timing is judged by the timing judgment device, the foreign matter judgment device outputs a judgment result performed at the output timing.

5. The determination system according to claim 4, wherein the timing determination device determines the processing timing based on an image captured by the imaging device or control information of the transport machine.

6. A judgment system according to any one of claims 1 to 5, wherein the foreign object judgment device acquires an image of the object to be judged taken from below of the iron scrap being lifted by the conveyor and makes a judgment.

7. A judgment system according to any one of claims 1 to 6, wherein the foreign object judgment device acquires an image of the object to be judged that is captured while the holding portion of the iron scrap being lifted by the conveyor is facing an imaging device, and makes a judgment based on that image.

8. A judgment system as claimed in any one of claims 1 to 7, wherein the foreign object judgment device acquires an image of the object to be judged, which is taken when the iron scrap that has been lifted at least once by the conveying machine is placed in an area different from the destination before being conveyed to the specified destination, and makes a judgment.

9. A method for determining whether or not the iron scrap contains a foreign object, comprising: an image capturing step of capturing an image by an imaging device provided on a conveying machine for transporting iron scrap; an acquisition step of acquiring the image captured by the imaging device as an image to be determined; and a foreign object determination step of determining whether or not the iron scrap contains a foreign object using the image to be determined acquired in the acquisition step.

10. The method according to claim 9, wherein in the image capturing step, an image is captured from below of the iron scrap being lifted by the conveyor.

11. A method for determining whether or not a scrap metal is to be scrapped, comprising: a step of: capturing an image of the scrap metal being scrapped; 12. A determination method as described in any one of claims 9 to 11, wherein in the image capturing step, an image is captured of the iron scrap that has been lifted at least once by the conveying machine while it is placed in an area different from the destination before being transported to a specified destination.

13. A computer program for causing a computer to function as a foreign matter determination device that acquires an image of an object to be determined captured by an imaging device provided on a conveying machine for transporting iron scrap, and determines whether or not the iron scrap contains a foreign matter using the acquired image of the object to be determined.