Foreign object detection device and foreign object detection method

Abstract

The foreign object detection device (10) is a foreign object detection device that detects foreign objects contained in scrap, and includes an image acquisition unit (131) that moves following a working unit that performs specified work on the scrap and acquires images taken by a camera that captures images of the surroundings of the working unit, and a determination unit (132) that detects potential foreign objects based on the images and determines the presence or absence of a foreign object based on the detection results of the potential foreign objects.

Description

[Technical field]

[0001] TECHNICAL FIELD The present disclosure relates to a foreign object detection device and a foreign object detection method, and more particularly to a foreign object detection device and a foreign object detection method for detecting foreign objects contained in scrap. [Background technology]

[0002] In the manufacture of steel products, it is desirable to increase the use of ferrous scrap as a raw material for iron in order to reduce the environmental impact. When melting scrap in the manufacture of steel products, it is necessary to remove foreign matter such as non-ferrous materials and sealed objects in advance. In order to remove foreign matter from scrap, it is necessary to determine whether or not there is foreign matter in the scrap.

[0003] In recent years, object recognition technology has been developed that uses deep learning to distinguish pre-learned objects from images of the target object. Object recognition technology can also be applied to detecting foreign objects from images of the target object, which is scrap. For example, Patent Document 1 proposes a system that uses a camera installed in a scrap yard to detect sealed objects contained in a collection of iron scrap. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-86285 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology of Patent Document 1 uses multiple cameras to capture images of iron scrap piled up in a scrap yard, iron scrap lifted by a crane, and iron scrap loaded onto a truck or ship. However, scrap yards are vast and contain multiple piles of scrap. Therefore, it is difficult to capture images of only the target scrap with a camera. In addition, when the target scrap is far from the camera, it is difficult to capture a high-precision image of the scrap, which reduces the accuracy of detecting foreign objects from the captured image.

[0006] An object of the present disclosure, made in consideration of the above circumstances, is to provide a foreign object detection device and a foreign object detection method that can identify foreign objects with a high degree of accuracy from a photographed image of a target scrap. [Means for solving the problem]

[0007] (1) A foreign object detection device according to an embodiment of the present disclosure, A foreign object detection device for detecting foreign objects contained in scrap, comprising: an image acquisition unit that moves following a working unit that performs a predetermined operation on the scrap and acquires images taken by a camera that captures images of the surroundings of the working unit; The image processing device further includes a determination unit that detects the candidate foreign object based on the image and determines the presence or absence of a foreign object based on a result of the detection of the candidate foreign object.

[0008] (2) As an embodiment of the present disclosure, in (1), The working unit is an arm of a heavy machine, and the camera is attached to the heavy machine or the arm.

[0009] (3) As an embodiment of the present disclosure, in (1) or (2), The image is a plurality of images captured by the camera with different fields of view, The determination unit detects the foreign object candidate in each of the plurality of images, and determines the presence or absence of the foreign object based on a detection result of the foreign object candidate in the plurality of images.

[0010] (4) As an embodiment of the present disclosure, in (3), The image acquisition unit acquires the images at time intervals of 30 seconds or less.

[0011] (5) A foreign object detection method according to an embodiment of the present disclosure, A foreign object detection method for detecting foreign objects contained in scrap, comprising the steps of: acquiring images captured by a camera that moves following a working unit that performs a predetermined operation on the scrap and captures images of the surroundings of the working unit; Detecting the candidate foreign object based on the image, and determining the presence or absence of a foreign object based on a result of detecting the candidate foreign object. Effect of the Invention

[0012] According to the present disclosure, it is possible to provide a foreign object detection device and a foreign object detection method that can identify foreign objects with high accuracy from a photographed image of a target scrap. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram showing an example of the configuration of a foreign object detection system including a foreign object detection device according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a flowchart showing the process of a foreign object detection method according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 is a diagram showing an example of N images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, a foreign object detection device 10 (see FIG. 1) and a foreign object detection method according to an embodiment of the present disclosure will be described with reference to the drawings.

[0015] (Anomaly detection system) Fig. 1 is a schematic diagram showing an example of the configuration of a foreign object detection system including a foreign object detection device 10 according to this embodiment. In Fig. 1, the foreign object detection device 10 is shown in a block diagram illustrating an example of the internal configuration. The foreign object detection device 10 is a device that detects foreign objects contained in scraps based on images of the scraps.

[0016] In the example of FIG. 1, the foreign object detection device 10 is used for inspecting iron-based scrap using heavy machinery with an arm. Iron-based scrap is used as an iron raw material in the manufacture of steel products. When melting scrap in the manufacture of steel products, it is necessary to remove foreign objects (non-iron materials and sealed objects, etc.) in advance. In order to remove foreign objects from the scrap, the foreign object detection device 10 judges whether or not there is a foreign object in the scrap. Here, a lifting magnet for transporting and moving the scrap is provided at the tip of the arm of the heavy machinery. The scrap is placed on the ground (yard) as a scrap pile. In order to detect and remove the foreign objects, a worker operating the heavy machinery with an arm performs an operation of spreading the scrap so that the pile is flat (hereinafter referred to as "leveling operation"). Here, the scrap pile means a collection of a lump of scrap piled up in a mountain shape. In the following explanation, the part that performs a predetermined operation on the scrap is referred to as the working unit. In the example of FIG. 1, the working unit is the arm of the heavy machinery that performs the leveling operation. Here, the working unit is not limited to the arm of the heavy equipment, and may be, for example, a lifting magnet that moves along a rail. Also, the predetermined work is not limited to the leveling work, and may be, for example, work such as transporting, moving, and lifting scrap. Also, in the example of Fig. 1, the worker is also a driver who operates the heavy equipment.

[0017] The foreign object detection device 10 includes a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 includes an image acquisition unit 131, a determination unit 132, and an output unit 133. The foreign object detection device 10 may be, for example, a computer as a hardware configuration. The computer may be a server computer or a portable computer such as a laptop or tablet. Details of the components of the foreign object detection device 10 will be described later. Here, the foreign object detection device 10 may not be a single device, but may be configured of multiple devices that are placed in multiple locations and can transmit and receive data to each other via a network. In other words, multiple devices connected by a network may function as the foreign object detection device 10 shown in FIG. 1 as a whole. Therefore, for example, the foreign object detection device 10 may be configured of one computer as a hardware configuration, or may be configured of multiple computers connected by a network. When configured by multiple computers, the storage unit 12 may be a shared memory that each computer can access. In this embodiment, the foreign object detection device 10 is described as a server computer installed at a location away from the yard where the scrap pile is placed, but it may be any type of computer installed in the yard. For example, as another configuration example, the foreign object detection device 10 may be a small computer built into a camera, or the foreign object detection device 10 may be integrated with a terminal device used by an operator.

[0018] The foreign object detection device 10 may constitute a foreign object detection system together with devices connected to the network. The network is, for example, the Internet. The network may be configured to include, for example, a LAN (Local Area Network) in part. In this embodiment, the foreign object detection system is configured to include a camera attached to a heavy machine or an arm. The camera moves following the working unit (arm) and captures images of the surroundings of the working unit. Since the heavy machine or arm moves relative to the scrap pile during work, the images captured by the camera also capture the scrap pile from multiple different directions (angles). In other words, the worker can capture the scrap pile from multiple different directions (angles) simply by operating the heavy machine to perform normal work (in other words, without the need to operate the camera separately). In addition, by installing the camera facing the lifting magnet as shown in FIG. 1, the surroundings of the tip of the arm can be reliably captured by the camera. Here, in this embodiment, the camera has a communication function and can output the captured image to the foreign object detection device 10 via the network. In addition, the camera may be configured to output the captured image to the foreign object detection device 10 via a terminal device.

[0019] Moreover, in this embodiment, the foreign object detection system includes a terminal device used by a worker performing work in the yard. The terminal device has at least a function of transmitting information regarding the presence or absence of a foreign object (specific examples include determination results, etc.) to the worker. The terminal device may be, but is not limited to, a general-purpose mobile terminal such as a smartphone or tablet. Here, the terminal device has a communication function and can obtain information regarding the presence or absence of a foreign object from the foreign object detection device 10 via a network.

[0020] As described above, the camera has a communication function, and takes an image of the scrap to be worked on during the leveling work, and outputs the image data to the foreign object detection device 10 via the network. In this embodiment, the camera takes multiple images of the scrap to be worked on with different fields of view (direction and angle of view) at a predetermined time interval. In this embodiment, the camera is installed on the arm of the heavy equipment, so the position and direction of the camera change in accordance with the operation of the position and direction of the arm by the worker during the leveling work. Here, the multiple images may be taken while spreading out the scrap pile, or may be taken while changing the field of view with the scrap pile not changing. In addition, as described later, in this embodiment, the determination result of the presence or absence of a foreign object based on the multiple images is notified to the terminal device used by the worker. Therefore, the worker who is also the driver of the heavy equipment can detect foreign objects with high accuracy without increasing the workload (i.e., by simply operating the heavy equipment as usual), and can efficiently inspect the iron-based scrap.

[0021] (Foreign object detection device) The components of the foreign object detection device 10 will be described in detail below. The communication unit 11 includes one or more communication modules that connect to a network. The communication unit 11 may include a communication module that supports mobile communication standards such as 4G (4th Generation) and 5G (5th Generation). The communication unit 11 may include a communication module that supports a wired or wireless LAN standard.

[0022] The storage unit 12 is one or more memories. The memory may be, for example, a semiconductor memory, a magnetic memory, an optical memory, or the like, but is not limited to these and may be any memory. The storage unit 12 is, for example, built into the foreign object detection device 10, but may also be configured to be accessed from the outside by the foreign object detection device 10 via any interface.

[0023] The storage unit 12 stores various data used in various calculations performed by the control unit 13. The storage unit 12 may also store results and intermediate data of various calculations performed by the control unit 13.

[0024] The storage unit 12 may temporarily store various information from devices connected via a network. In this embodiment, the storage unit 12 stores multiple images of the scrap pile with different fields of view captured by a camera at a predetermined time interval. In this embodiment, the storage unit 12 stores the presence or absence of a foreign object candidate determined by the determination unit 132 in association with each of the multiple images. The storage unit 12 may also store the presence or absence and position of a foreign object determined by the determination unit 132. Here, a foreign object candidate means a foreign object determined in one image. As will be described later, in this embodiment, the final presence or absence of a foreign object is determined in multiple images. Therefore, in order to distinguish from the final foreign object, a foreign object determined in one image is called a "foreign object candidate."

[0025] The control unit 13 is one or more processors. The processor may be, for example, a general-purpose processor or a dedicated processor specialized for a specific process, but is not limited to these and may be any processor. The control unit 13 controls the overall operation of the foreign object detection device 10.

[0026] Here, the foreign object detection device 10 may have the following software configuration. One or more programs used to control the operation of the foreign object detection device 10 are stored in the storage unit 12. When the program stored in the storage unit 12 is read by the processor of the control unit 13, it causes the control unit 13 to function as an image acquisition unit 131, a determination unit 132, and an output unit 133.

[0027] The image acquisition unit 131 acquires an image captured by a camera. In this embodiment, the image acquisition unit 131 acquires a plurality of images captured by a camera with different fields of view. Here, the interval (acquisition interval) at which the image acquisition unit 131 acquires an image may be the above-mentioned predetermined time interval, i.e., the shooting interval of the camera, but may be longer than the predetermined time interval. In order to efficiently acquire a plurality of images with different fields of view, the acquisition interval of the image acquisition unit 131 may be set based on the moving speed of the heavy machine or the operating speed of the arm. However, it is preferable that the acquisition interval is short enough that there is no change in the target scrap pile in the predetermined work (leveling work in this embodiment). For example, the acquisition interval is within 30 seconds. This is because if the acquisition interval is too long, the heavy machine may move to another scrap pile in the middle, and the scrap pile photographed in the acquired plurality of images may be different. Therefore, it is preferable that the image acquisition unit 131 acquires a plurality of images at a time interval of 30 seconds or less.

[0028] The determination unit 132 detects possible foreign objects based on the images acquired by the image acquisition unit 131, and determines the presence or absence of a foreign object in the scrap pile based on the detection results of the possible foreign objects. In this embodiment, the number of images acquired by the image acquisition unit 131 is at least N (N is an integer equal to or greater than 2). The determination unit 132 detects possible foreign objects in each of the multiple images, and determines the presence or absence of a foreign object based on the detection results of the possible foreign objects in the multiple images. In addition to the presence or absence of a foreign object, the position of the foreign object may also be determined. Details of the determination of the presence or absence of a foreign object will be described later.

[0029] The output unit 133 outputs information on the presence or absence of a foreign object, including the determination result by the determination unit 132, to a terminal device or the like. The information on the presence or absence of a foreign object may be audio, image, or text information. If the terminal device is equipped with, for example, a speaker, and can convey information to the worker by audio, the output unit 133 may output audio information on the presence or absence of a foreign object. If the terminal device is equipped with, for example, a display, and can convey information to the worker by image, the output unit 133 may output image information on the presence or absence of a foreign object. The output unit 133 may also output a control signal to the terminal device to perform a predetermined operation according to the presence or absence of a foreign object. For example, if the determination unit 132 determines that a foreign object is present, the output unit 133 may output a control signal to the terminal device to generate an alarm sound. The output unit 133 may also send an email or the like including the determination result to the terminal device or a smartphone of a yard manager or the like.

[0030] (Foreign object detection method) 2 is a flowchart showing the process of the foreign object detection method according to this embodiment. The process of the foreign object detection method is executed by the foreign object detection device 10, and may be started, for example, when at least N images have been captured after the camera has started capturing images.

[0031] The image acquisition unit 131 acquires data of an image captured by a camera (step S1). The determination unit 132 determines whether or not a candidate for a foreign object is present in the image acquired by the image acquisition unit 131 (step S2). Then, the determination result by the determination unit 132 (presence or absence of a candidate for a foreign object) is recorded (step S3). In other words, the presence or absence of a candidate for a foreign object is associated with the acquired image and stored in the storage unit 12.

[0032] Here, the detection of a candidate foreign object by the determination unit 132 may be performed using a known method for object recognition in an image. For example, a model generated by machine learning using past images including foreign objects and images not including foreign objects as learning data may be used. As a specific example, a pre-trained deep learning model may be used. Furthermore, the position of a foreign object may also be determined using such a model.

[0033] In this embodiment, the determination unit 132 determines whether or not there is a foreign object using not only one image (the most recently obtained image) but N (multiple) images. As described above, the determination results (whether or not there is a foreign object candidate) by the determination unit 132 are stored in the storage unit 12. The determination unit 132 acquires the determination results of the past N images including the most recent image from the storage unit 12. The determination unit 132 then calculates the number of times that it has been determined that there is a foreign object candidate in the N images (step S4). In the following description, it is assumed that the number of times that it has been determined that there is a foreign object candidate in step S4 is k. The determination unit 132 also calculates the ratio (k / N) of times that it has been determined that there is a foreign object candidate.

[0034] FIG. 3 is a diagram showing an example of N images. The time when the most recent image was obtained is "T", and the above-mentioned acquisition interval is "ΔT". The determination unit 132 obtains from the storage unit 12 the presence or absence of a foreign object candidate for the past N images up to T-(N-1)×ΔT. In the example of FIG. 3, the first, second, and fourth images are determined to have a foreign object candidate "present". Also, in the example of FIG. 3, the third and Nth images are determined to have a foreign object candidate "absent". Here, if N is 5 in the example of FIG. 3, the number of times it was determined that a foreign object candidate was present was 3, and the above-mentioned ratio (k / N) is 60%.

[0035] The determination unit 132 determines whether the ratio (k / N) of the ratios determined to have a foreign object candidate is equal to or greater than a threshold (step S5). If the ratio (k / N) is equal to or greater than the threshold (Yes in step S5), the determination unit 132 determines that the target scrap pile has a foreign object (step S6). If the ratio (k / N) is less than the threshold (No in step S5), the determination unit 132 determines that the target scrap pile does not have a foreign object (step S7). The determination result may be output to a terminal device by the output unit 133. Here, the threshold is 50% as an example, but is not limited to this, and may be changed according to, for example, the value of N. As a specific example, it is assumed that the interval (acquisition interval) at which the image acquisition unit 131 acquires images, i.e., ΔT, is 10 seconds, the threshold is 50%, and N is 5. In the example of FIG. 3, the ratio (k / N) is 60%, which is equal to or greater than the threshold, and therefore the determination unit 132 determines that a foreign object is present.

[0036] According to the foreign object detection method of this embodiment, multiple images with different fields of view are taken of the same scrap, and foreign object detection is performed based on the multiple images. Therefore, foreign objects that are difficult to see from one direction can be detected in an image seen from another direction, so that overlooking can be prevented. In addition, since foreign object detection is performed based on multiple images, erroneous detection of objects other than foreign objects as foreign objects can be prevented. In addition, since foreign object detection is possible with only multiple images from the camera, the processing load is reduced, and the time until the judgment result is obtained can be shortened. Since the worker can obtain the judgment result almost in real time, it is possible to efficiently proceed with inspection work, etc.

[0037] As described above, the foreign object detection device 10 and the foreign object detection method according to the present embodiment can accurately identify foreign objects from a captured image of the target scrap. In addition, it is possible to prevent quality degradation of steel products manufactured by melting scrap due to the inclusion of impurity elements. It is also possible to prevent steam explosions and the like caused by the inclusion of sealed objects in the manufacture of steel products.

[0038] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions included in each component or each step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined into one or divided. The embodiments of the present disclosure can also be realized as a program executed by a processor included in the device or a storage medium on which a program is recorded. It should be understood that these are also included in the scope of the present disclosure.

[0039] In the above embodiment, the camera was attached to the arm of the heavy equipment. Here, the camera may be attached to a location other than the arm of the heavy equipment (for example, above the driver's seat, etc.). Even if the camera is attached to a location other than the arm, the camera moves together with the heavy equipment when the heavy equipment moves forward, backward, and turns, so that the camera can always photograph the tip of the arm. Also, the camera does not need to be attached to the heavy equipment as long as it can follow the movement of the arm. Specifically, the camera may be attached to, for example, an unmanned aerial vehicle, and the unmanned aerial vehicle may fly around the heavy equipment so that the camera is always facing the tip of the arm. [Explanation of symbols]

[0040] 10 Foreign object detection device 11 Communications Department 12 Storage section 13 Control section 131 Image acquisition unit 132 Judgment section 133 Output section

Claims

1. A foreign object detection device for detecting foreign objects contained in scrap stored on the ground as a scrap pile, comprising: an image acquisition unit that moves following a working unit that performs a predetermined operation on the scrap and acquires images taken by a camera that captures images of the surroundings of the working unit; a determination unit that detects the candidate foreign object based on the image and determines the presence or absence of a foreign object based on a detection result of the candidate foreign object, The working unit is an arm of a heavy machine that performs a leveling operation to spread the scrap so that the scrap pile is flat, and the camera is attached to the heavy machine or the arm, the images being a plurality of images with different fields of view taken by the camera while the arm is spreading the scrap pile or while the scrap pile is stationary; The determination unit detects the foreign object candidates in each of the plurality of images, and determines the presence or absence of the foreign object based on a proportion of the foreign object candidates calculated based on a detection result of the foreign object candidates in the plurality of images.

2. The foreign object detecting device according to claim 1 , wherein the image acquiring unit acquires the plurality of images at time intervals of 30 seconds or less.

3. A method for detecting foreign objects contained in scrap stored on the ground as a scrap pile, comprising: acquiring images captured by a camera that moves following a working unit that performs a predetermined operation on the scrap and captures images of the surroundings of the working unit; detecting the candidate foreign object based on the image, and determining the presence or absence of a foreign object based on a result of detecting the candidate foreign object; The working unit is an arm of a heavy machine that performs a leveling operation to spread the scrap so that the scrap pile is flat, and the camera is attached to the heavy machine or the arm, the images being a plurality of images with different fields of view taken by the camera while the arm is spreading the scrap pile or while the scrap pile is stationary; a ratio of the foreign object candidates calculated based on a detection result of the foreign object candidates in the plurality of images, the ratio of the foreign object candidates calculated based on a detection result of the foreign object candidates in the plurality of images, and a determination of the presence or absence of the foreign object.