Foreign matter detection device and foreign matter detection method

Abstract

A foreign matter detection device (10) for detecting foreign matter contained in scrap, said device comprising: an image acquisition unit (131) for acquiring images captured by a camera that moves so as to follow a work unit performing prescribed work on the scrap and captures images of the surroundings of the work unit; and a determination unit (132) for detecting candidates for the foreign matter on the basis of the images and determining the presence / absence of foreign matter on the basis of detection results for the candidates for the foreign matter.

Description

Foreign object detection device and foreign object detection method

[0001] TECHNICAL FIELD The present disclosure relates to a foreign object detection device and a foreign object detection method, and more particularly to a foreign object detection device and a foreign object detection method for detecting foreign objects contained in scrap.

[0002] In the manufacture of steel products, it is desirable to increase the use of iron-based scrap as a raw material to reduce the environmental impact. When scrap is melted in the manufacture of steel products, it is necessary to remove foreign matter such as non-iron materials and sealed objects beforehand. In order to remove foreign matter from the scrap, it is necessary to determine whether or not there is any foreign matter in the scrap.

[0003] In recent years, object recognition technology has been developed that uses deep learning to identify pre-learned objects from photographed images of the target object. Object recognition technology can also be applied to detecting foreign objects from images of scrap, which is the target object. For example, Patent Document 1 proposes a system that uses a camera installed in a scrap yard to detect sealed objects contained in a collection of iron scrap.

[0004] Japanese Patent Application Laid-Open No. 2021-86285

[0005] The technology of Patent Document 1 uses multiple cameras to capture images of scrap iron piled up in a scrap yard, scrap iron lifted by a crane, and scrap iron loaded onto a truck or ship. However, scrap yards are large and contain multiple piles of scrap. This makes it difficult to capture images of only the target scrap with a camera. Furthermore, if the target scrap is far from the camera, it is difficult to capture high-precision images of the scrap, which reduces the accuracy of detecting foreign objects from the captured images.

[0006] In view of the above circumstances, an object of the present disclosure is to provide a foreign matter detection device and a foreign matter detection method that can identify foreign matter with high accuracy from a photographed image of the target scrap.

[0007] (1) A foreign matter detection device according to one embodiment of the present disclosure is a foreign matter detection device that detects foreign matter contained in scrap, and includes: an image acquisition unit that moves in accordance with a work unit that performs a predetermined task on the scrap and acquires images taken by a camera that captures images of the surroundings of the work unit; and a determination unit that detects potential foreign matter based on the images and determines whether or not there is a foreign matter based on the detection results of the potential foreign matter.

[0008] (2) As one embodiment of the present disclosure, in (1), the working unit is an arm of a heavy machine, and the camera is attached to the heavy machine or the arm.

[0009] (3) As an embodiment of the present disclosure, in (1) or (2), the images are a plurality of images captured by the camera with different fields of view, and the determination unit detects the foreign object candidate in each of the plurality of images and determines the presence or absence of the foreign object based on the detection results of the foreign object candidate in the plurality of images.

[0010] (4) As an embodiment of the present disclosure, in (3), the image acquisition unit acquires the plurality of images at time intervals of 30 seconds or less.

[0011] (5) A foreign object detection method according to one embodiment of the present disclosure is a foreign object detection method for detecting foreign objects contained in scrap, comprising: acquiring images captured by a camera that moves in accordance with a working unit that performs a predetermined task on the scrap and captures images of the surroundings of the working unit; detecting potential foreign objects based on the images; and determining the presence or absence of foreign objects based on the detection results of the potential foreign objects.

[0012] According to the present disclosure, it is possible to provide a foreign matter detection device and a foreign matter detection method that can identify foreign matter with high accuracy from a photographed image of the target scrap.

[0013] Fig. 1 is a schematic diagram showing an example of the configuration of a foreign object detection system including a foreign object detection device according to an embodiment of the present disclosure. Fig. 2 is a flowchart showing the processing of a foreign object detection method according to an embodiment of the present disclosure. Fig. 3 is a diagram showing an example of N images.

[0014] Hereinafter, a foreign object detection device 10 (see FIG. 1) and a foreign object detection method according to an embodiment of the present disclosure will be described with reference to the drawings.

[0015] (Anomaly Detection System) Fig. 1 is a schematic diagram showing an example of the configuration of a foreign object detection system including a foreign object detection device 10 according to this embodiment. In Fig. 1, the foreign object detection device 10 is shown in a block diagram illustrating an example of the internal configuration. The foreign object detection device 10 is a device that detects foreign objects contained in scrap based on images of the scrap.

[0016] In the example shown in FIG. 1 , the foreign object detection device 10 is used to inspect ferrous scrap using heavy machinery with an arm. Ferrous scrap is used as a raw material in the manufacture of steel products. When melting scrap in the manufacture of steel products, foreign objects (such as non-ferrous materials and sealed objects) must be removed beforehand. To remove foreign objects from the scrap, the foreign object detection device 10 determines whether or not the scrap contains any foreign objects. A lifting magnet is attached to the tip of the arm of the heavy machinery for transporting and moving the scrap. The scrap is stored as a pile on the ground (yard). To detect and remove foreign objects, the operator of the heavy machinery with an arm spreads the scrap to flatten the pile (hereinafter referred to as "leveling work"). Here, a "scrap pile" refers to a collection of scrap piled up in a mountain shape. In the following description, the part that performs the specified work on the scrap is referred to as the "working unit." In the example shown in FIG. 1 , the working unit is the arm of the heavy machinery that performs the leveling work. Here, the working unit is not limited to the arm of the heavy equipment, but may be, for example, a lifting magnet that moves along a rail. Also, the predetermined work is not limited to leveling work, but may be work such as transporting, moving, or lifting scrap. In the example of Figure 1, the worker is also the driver who operates the heavy equipment.

[0017] The foreign object detection device 10 includes a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 includes an image acquisition unit 131, a determination unit 132, and an output unit 133. The foreign object detection device 10 may be, for example, a computer in its hardware configuration. The computer may be a server computer or a portable computer such as a laptop or tablet. Details of the components of the foreign object detection device 10 will be described later. The foreign object detection device 10 may not be a single device, but may be composed of multiple devices located in multiple locations and capable of transmitting and receiving data to and from each other via a network. In other words, multiple devices connected via a network may function as the foreign object detection device 10 shown in FIG. 1 . Therefore, for example, the foreign object detection device 10 may be, for example, a single computer in its hardware configuration, or multiple computers connected via a network. When the foreign object detection device 10 is composed of multiple computers, the storage unit 12 may be a shared memory accessible by each computer. In this embodiment, the foreign object detection device 10 is described as a server computer installed at a location away from the yard where the scrap pile is placed, but it may be any type of computer installed in the yard. For example, in another configuration example, the foreign object detection device 10 may be a small computer built into a camera, or the foreign object detection device 10 may be integrated into a terminal device used by an operator.

[0018] The foreign object detection device 10, together with devices connected via a network, may constitute a foreign object detection system. The network may be, for example, the Internet. The network may also include, for example, a local area network (LAN). In this embodiment, the foreign object detection system includes a camera attached to a heavy machine or an arm. The camera follows the working unit (arm) and captures images of the surrounding area. Because the heavy machine or arm moves relative to the scrap pile during operation, the images captured by the camera capture the scrap pile from multiple different directions (angles). In other words, a worker can capture images of the scrap pile from multiple different directions (angles) simply by operating the heavy machine (i.e., without having to operate the camera separately). Furthermore, by arranging the camera facing the lifting magnet as shown in FIG. 1 , the camera can reliably capture images of the area around the tip of the arm. In this embodiment, the camera has a communication function and can output captured images to the foreign object detection device 10 via the network. The camera may also be configured to output the captured image to the foreign object detection device 10 via a terminal device.

[0019] In this embodiment, the foreign object detection system includes a terminal device used by a worker performing work in the yard. The terminal device has at least a function for transmitting information regarding the presence or absence of a foreign object (specifically, a determination result, etc.) to the worker. The terminal device may be, but is not limited to, a general-purpose mobile terminal such as a smartphone or tablet. Here, the terminal device has a communication function and can obtain information regarding the presence or absence of a foreign object from the foreign object detection device 10 via a network.

[0020] As described above, the camera has a communication function and captures images of the scrap being worked on during the leveling operation and outputs the image data to the foreign object detection device 10 via a network. In this embodiment, the camera captures multiple images of the scrap being worked on at predetermined time intervals, each image capturing different fields of view (orientation and angle of view). In this embodiment, the camera is attached to the arm of the heavy equipment, and the camera's position and orientation change as the worker manipulates the arm's position and orientation during the leveling operation. Here, multiple images may be captured while the scrap pile is being expanded, or may be captured while the field of view is changed while the scrap pile remains stationary. Furthermore, as described below, in this embodiment, the determination of the presence or absence of foreign objects based on the multiple images is notified to a terminal device used by the worker. Therefore, the worker, who also operates the heavy equipment, can detect foreign objects with high accuracy without increasing his or her workload (i.e., simply by operating the heavy equipment as usual), allowing for efficient inspection of ferrous scrap.

[0021] (Foreign object detection device) The components of the foreign object detection device 10 will be described in detail below. The communication unit 11 is configured to include one or more communication modules that connect to a network. The communication unit 11 may include a communication module that supports mobile communication standards such as 4G (4th Generation) and 5G (5th Generation). The communication unit 11 may include a communication module that supports a wired or wireless LAN standard, for example.

[0022] The storage unit 12 is one or more memories. The memory may be, for example, a semiconductor memory, a magnetic memory, an optical memory, or the like, but is not limited to these, and may be any memory. The storage unit 12 is, for example, built into the foreign object detection device 10, but may also be configured to be accessed from outside by the foreign object detection device 10 via any interface.

[0023] The storage unit 12 stores various data used in various calculations performed by the control unit 13. The storage unit 12 may also store results and intermediate data of various calculations performed by the control unit 13.

[0024] The storage unit 12 may temporarily store various information from devices connected via a network. In this embodiment, the storage unit 12 stores multiple images of a scrap pile with different fields of view captured by a camera at predetermined time intervals. In this embodiment, the storage unit 12 also stores the presence or absence of a foreign object candidate determined by the determination unit 132, linking it to each of the multiple images. The storage unit 12 may also store the presence or absence and position of a foreign object determined by the determination unit 132. Here, a foreign object candidate refers to a foreign object determined in a single image. As will be described later, in this embodiment, the final presence or absence of a foreign object is determined based on multiple images. Therefore, to distinguish a foreign object determined in a single image from a final foreign object, the foreign object determined in a single image is referred to as a "foreign object candidate."

[0025] The control unit 13 is one or more processors. The processor may be, for example, a general-purpose processor or a dedicated processor specialized for a specific process, but is not limited to these and may be any processor. The control unit 13 controls the overall operation of the foreign object detection device 10.

[0026] Here, the foreign object detection device 10 may have the following software configuration: One or more programs used to control the operation of the foreign object detection device 10 are stored in the storage unit 12. When the programs stored in the storage unit 12 are read by the processor of the control unit 13, they cause the control unit 13 to function as an image acquisition unit 131, a determination unit 132, and an output unit 133.

[0027] The image acquisition unit 131 acquires images captured by a camera. In this embodiment, the image acquisition unit 131 acquires multiple images with different fields of view captured by the camera. Here, the interval (acquisition interval) at which the image acquisition unit 131 acquires images may be the above-mentioned predetermined time interval, i.e., the camera's shooting interval, or may be longer than the predetermined time interval. To efficiently acquire multiple images with different fields of view, the acquisition interval of the image acquisition unit 131 may be set based on the movement speed of the heavy equipment or the operation speed of the arm. However, it is preferable that the acquisition interval be short enough to prevent changes in the target scrap pile during the predetermined work (leveling work in this embodiment). For example, the acquisition interval is within 30 seconds. This is because if the acquisition interval is too long, the heavy equipment may move to a different scrap pile during the work, and the scrap pile captured in the multiple acquired images may be different. Therefore, it is preferable that the image acquisition unit 131 acquires multiple images at time intervals of 30 seconds or less.

[0028] The determination unit 132 detects potential foreign objects based on the images acquired by the image acquisition unit 131, and determines the presence or absence of foreign objects in the scrap pile based on the detection results of the potential foreign objects. In this embodiment, the number of images acquired by the image acquisition unit 131 is at least N (N is an integer equal to or greater than 2). The determination unit 132 detects potential foreign objects in each of the multiple images, and determines the presence or absence of foreign objects based on the detection results of the potential foreign objects in the multiple images. In addition to the presence or absence of foreign objects, the position of the foreign objects may also be determined. Details of the determination of the presence or absence of foreign objects will be described later.

[0029] The output unit 133 outputs information regarding the presence or absence of a foreign object, including the determination result by the determination unit 132, to a terminal device or the like. The information regarding the presence or absence of a foreign object may be audio, image, or text information. If the terminal device is equipped with, for example, a speaker or the like and can convey information to the worker audio-wise, the output unit 133 may output audio information regarding the presence or absence of a foreign object. If the terminal device is equipped with, for example, a display or the like and can convey information to the worker visually, the output unit 133 may output image information regarding the presence or absence of a foreign object. Furthermore, the output unit 133 may output a control signal to the terminal device to cause the terminal device to perform a predetermined operation depending on the presence or absence of a foreign object. For example, if the determination unit 132 determines that a foreign object is present, the output unit 133 may output a control signal to the terminal device to cause the terminal device to emit an alarm. Furthermore, the output unit 133 may send an email or the like including the determination result to the terminal device or a smartphone of the yard manager or the like.

[0030] 2 is a flowchart showing the process of the foreign object detection method according to this embodiment. The process of the foreign object detection method is executed by the foreign object detection device 10, and may be started, for example, when at least N images have been captured after the camera starts capturing images.

[0031] The image acquisition unit 131 acquires data of an image captured by a camera (step S1). The determination unit 132 determines whether or not a foreign object candidate exists in the image acquired by the image acquisition unit 131 (step S2). The determination result by the determination unit 132 (whether or not a foreign object candidate exists) is then recorded (step S3). That is, the presence or absence of a foreign object candidate is linked to the acquired image and stored in the storage unit 12.

[0032] Here, the detection of foreign object candidates by the determination unit 132 may be performed using a known method for object recognition in images. For example, a model generated by machine learning using past images containing foreign objects and past images not containing foreign objects as training data may be used. As a specific example, a pre-trained deep learning model may be used. Furthermore, the position of the foreign object may also be determined using such a model.

[0033] In this embodiment, the determination unit 132 determines whether or not a foreign substance is present not only using one image (the most recently acquired image) but also using N (multiple) images. As described above, the determination results (presence or absence of a foreign substance candidate) by the determination unit 132 are stored in the storage unit 12. The determination unit 132 acquires the determination results of the past N images, including the most recent image, from the storage unit 12. The determination unit 132 then calculates the number of times that a foreign substance candidate is determined to be present in the N images (step S4). In the following description, it is assumed that the number of times that a foreign substance candidate is determined to be present in step S4 is k. The determination unit 132 also calculates the ratio (k / N) of the number of times that a foreign substance candidate is determined to be present.

[0034] FIG. 3 is a diagram showing an example of N images. The time when the most recent image was acquired is "T," and the acquisition interval is "ΔT." The determination unit 132 acquires from the storage unit 12 the presence or absence of foreign substance candidates for the past N images up to T-(N-1)×ΔT. In the example of FIG. 3, it is determined that foreign substance candidates are "present" in the first, second, and fourth images. Also, in the example of FIG. 3, it is determined that foreign substance candidates are "absent" in the third and Nth images. Here, if N is 5 in the example of FIG. 3, the number of times that it was determined that foreign substance candidates were present is three, and therefore the above-mentioned ratio (k / N) is 60%.

[0035] The determination unit 132 determines whether the ratio (k / N) of the number of foreign matter candidates determined to exist is equal to or greater than a threshold (step S5). If the ratio (k / N) is equal to or greater than the threshold (Yes in step S5), the determination unit 132 determines that a foreign matter exists in the target scrap pile (step S6). If the ratio (k / N) is less than the threshold (No in step S5), the determination unit 132 determines that a foreign matter does not exist in the target scrap pile (step S7). The determination result may be output to a terminal device by the output unit 133. Here, the threshold is 50%, for example, but is not limited thereto and may be changed depending on, for example, the value of N. As a specific example, assume that the interval (acquisition interval) at which the image acquisition unit 131 acquires images, i.e., ΔT, is 10 seconds, the threshold is 50%, and N is 5. In the example of FIG. 3, the ratio (k / N) is 60%, which is equal to or greater than the threshold, and therefore the determination unit 132 determines that a foreign matter exists.

[0036] According to the foreign object detection method of this embodiment, multiple images of the same scrap with different fields of view are captured, and foreign object detection is performed based on the multiple images. Therefore, foreign objects that are difficult to see from one direction can be detected in an image viewed from another direction, preventing them from being overlooked. Furthermore, because foreign object detection is performed based on multiple images, false positives, such as detecting something other than a foreign object, can be prevented. Furthermore, because foreign object detection is possible using only multiple images from the camera, the processing load is reduced, shortening the time it takes to obtain a judgment result. Since workers can obtain judgment results almost in real time, they can efficiently carry out inspection work, etc.

[0037] As described above, the foreign object detection device 10 and foreign object detection method according to this embodiment can accurately identify foreign objects from photographed images of target scrap. Furthermore, it is possible to prevent quality degradation due to the inclusion of impurity elements in steel products manufactured by melting scrap. Furthermore, it is possible to prevent steam explosions and other problems caused by the inclusion of sealed objects in the steel product manufacturing process.

[0038] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art would easily be able to make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included within the scope of the present disclosure. For example, functions included in each component or step can be rearranged so as not to cause logical inconsistencies, and multiple components or steps can be combined or divided into one. The embodiments of the present disclosure can also be realized as a program executed by a processor included in an apparatus or a storage medium on which a program is recorded. It should be understood that these are also included within the scope of the present disclosure.

[0039] In the above embodiment, the camera is attached to the arm of the heavy equipment. However, the camera may be attached somewhere other than the arm of the heavy equipment (for example, above the driver's seat, etc.). Even if the camera is attached somewhere other than the arm, the camera moves with the heavy equipment when the heavy equipment moves forward, backward, and turns, so that the camera can always capture the tip of the arm. Furthermore, the camera does not need to be attached to the heavy equipment as long as it can follow the movement of the arm. Specifically, for example, the camera may be attached to an unmanned aerial vehicle, and the unmanned aerial vehicle may fly around the heavy equipment so that the camera is always facing the tip of the arm.

[0040] REFERENCE SIGNS LIST 10 Foreign object detection device 11 Communication unit 12 Storage unit 13 Control unit 131 Image acquisition unit 132 Determination unit 133 Output unit

Claims

1. A foreign object detection device that detects foreign objects contained in scrap, comprising: an image acquisition unit that moves in accordance with a working unit that performs specified work on the scrap and acquires images taken by a camera that captures images of the surroundings of the working unit; and a determination unit that detects potential foreign objects based on the images and determines the presence or absence of foreign objects based on the detection results of the potential foreign objects.

2. The foreign object detection device according to claim 1, wherein the working unit is an arm of heavy equipment, and the camera is attached to the heavy equipment or the arm.

3. A foreign object detection device as described in claim 1 or 2, wherein the images are multiple images captured by the camera with different fields of view, and the judgment unit detects the foreign object candidate in each of the multiple images and judges the presence or absence of the foreign object based on the detection results of the foreign object candidate in the multiple images.

4. The foreign object detection device according to claim 3, wherein the image acquisition unit acquires the plurality of images at time intervals of 30 seconds or less.

5. A foreign object detection method for detecting foreign objects contained in scrap, comprising: acquiring images taken by a camera which moves in accordance with a working unit which performs a specified operation on the scrap and takes images of the surroundings of the working unit; detecting potential foreign objects based on the images; and determining the presence or absence of foreign objects based on the detection results of the potential foreign objects.

Images