Determining system, determining method and computer program

Abstract

A determination system includes a camera provided to a conveyer for conveying scrap iron, and a foreign matter determination device that acquires a determination target image captured by the camera and determines whether or not the scrap iron contains foreign matter.

Description

Technical Field

[0001] This invention relates to a determination system, a determination method, and a computer program.

[0002] This application claims priority based on Japanese Patent Application No. 2023-190706, filed in Japan on November 8, 2023, the contents of which are incorporated herein by reference. Background Technology

[0003] In recent years, the steel industry has seen increased demands for recycled iron, aiming to reduce carbon dioxide emissions and other effects. The types of scrap iron (recycled iron) included in recycling are diverse. For example, new chips generated during steel product processing are classified as high-grade chips, while H2 and other pollutants generated from building demolition are primarily classified as low-grade chips. To manufacture high-grade steel, it is preferable to use high-quality high-grade chips; however, using only high-grade chips may lead to insufficient future procurement. Therefore, it is necessary to make full use of low-grade chips.

[0004] The high proportion of impurities other than iron in low-grade scrap metal leads to significant deviations. This has a substantial impact on the contamination of prohibited items such as motors or switchboards in scrap metal. If elements that are difficult to remove during the steelmaking process (residual elements: TE: tramp element, representative elements: Cu, Sn, Ni, Cr, Mo) are mixed into the steel, they will become a major cause of cracks and substandard material properties (mechanical properties) during hot rolling. To address this issue, Patent Document 1 discloses a specific example of a scrap metal handling device using a bridge crane to detect prohibited items along its path. Specifically, a bridge crane is pre-installed, and scrap metal is transported by the bridge crane. The movement path of the bridge crane is pre-fixed, and a camera is pre-installed along this path. When the bridge crane is used to transport scrap metal, the camera along its path photographs the scrap metal. Using the images captured by the camera, the contamination of prohibited items is determined, prompting the operator to remove them.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2020-176909 Summary of the Invention

[0008] The technical problem that the invention aims to solve

[0009] As described above, existing technologies are based on systems where the movement path of scrap metal handling equipment is fixed. On the other hand, the use of heavy machinery in scrap metal handling is considered. The purchase of heavy machinery has the advantage of reducing initial investment compared to the construction of bridge cranes. Furthermore, while bridge cranes have a fixed movement path, heavy machinery offers the advantage of free movement, meaning the scrap metal handling path is not fixed. Because it can move freely, unloading and loading can be performed at any location within the scrap metal storage area. Therefore, compared to situations where the location of the scrap metal storage area is limited to areas accessible by bridge cranes, the degree of freedom in utilizing the scrap metal storage area is significantly increased.

[0010] However, if the cameras are fixed to the ground, walls, or other fixed objects as before, multiple fixed cameras need to be installed throughout the entire operating area of ​​heavy machinery, which expands significantly with the increased freedom of the conveying path. This increases the cost of camera installation. To reduce the number of camera installation locations, the conveying path of the heavy machinery can be determined by moving scrap metal near the cameras. However, maintaining a constant distance between the camera and the scrap metal increases the burden on the heavy machinery operator, which is not simple. Therefore, the distance between the scrap metal and the camera varies each time a shot is taken, potentially reducing detection accuracy. Furthermore, it may also compromise the freedom of the conveying path.

[0011] The same issues arise when the camera is fixed to a stationary object, even in existing technologies using bridge cranes. Furthermore, this problem is not limited to the detection of prohibited substances containing residual elements; it is a common problem for all foreign objects that should be detected in other situations.

[0012] Therefore, in view of the above circumstances, the present invention provides a technology that can accurately determine whether foreign objects mixed into scrap iron are present when a conveying machine is used in the handling of scrap iron.

[0013] Technical solutions for solving technical problems

[0014] (1) One aspect of the present invention is a determination system, comprising: a shooting device disposed on a conveyor for transporting scrap iron; and a foreign object determination device that acquires an image of the object to be determined by the shooting device and determines whether the scrap iron contains foreign objects.

[0015] (2) In one aspect of the present invention, in the determination system described in (1) above, the shooting device is disposed on the main body or movable body of the transporter.

[0016] (3) In one aspect of the present invention, in the determination system described in (1) or (2) above, the shooting device takes pictures of the scrap iron during the transportation process of the transporter.

[0017] (4) In one embodiment of the present invention, the determination system described in any one of (1) to (3) above further includes a timing determination device, which determines the processing timing. The processing timing is any timing among the following: image acquisition timing for acquiring the image of the determination object, shooting timing for capturing the image of the determination object, or output timing for outputting the determination result of the foreign object determination device. When the timing determination device determines that the image acquisition timing is determined, the foreign object determination device acquires the image captured by the shooting device at the image acquisition timing. When the timing determination device determines that the shooting timing is determined, the shooting device performs the shooting at the shooting timing. When the timing determination device determines that the output timing is determined, the foreign object determination device outputs the determination result performed at the output timing.

[0018] (5) In one aspect of the present invention, in the determination system described in (4) above, the timing determination device determines the processing timing based on the image captured by the imaging device or the control information of the conveyor.

[0019] (6) In one aspect of the present invention, in the determination system described in any one of (1) to (5) above, the foreign object determination device obtains an image of the object to be determined by taking a picture of the scrap iron in the state of being lifted by the conveyor from below and makes a determination.

[0020] (7) In one aspect of the present invention, in the determination system described in any one of (1) to (6) above, the foreign object determination device acquires an image of the object to be determined while the holding portion of the scrap iron lifted by the conveyor is facing the imaging device, and makes a determination.

[0021] (8) In one aspect of the present invention, in the determination system of any one of (1) to (7) above, the foreign object determination device acquires and determines the image of the object to be determined, wherein the image of the object to be determined is an image taken in a state where the scrap iron, which is lifted at least once by the transporter, is placed in an area different from the transport destination before being transported to the designated transport destination.

[0022] (9) One aspect of the present invention is a determination method, comprising: an image capturing step, wherein an image is captured by a capturing device provided on a conveyor for transporting scrap iron; an acquisition step, wherein the image captured by the capturing device is acquired as an image to be determined; and a foreign object determination step, wherein the image to be determined acquired in the acquisition step is used to determine whether the scrap iron contains foreign objects.

[0023] (10) In one aspect of the present invention, in the determination method described in (9) above, an image of scrap iron in a state being lifted by the conveyor is captured from below during the image capturing step.

[0024] (11) In one aspect of the present invention, in the determination method described in (9) or (10) above, during the image capturing step, an image is captured while the holding portion of the scrap iron lifted by the conveyor is facing the capturing device.

[0025] (12) In one aspect of the present invention, in the determination method of any one of (9) to (11) above, during the image capturing step, an image is captured while the scrap iron, which is lifted at least once by the transporter, is placed in an area different from the transport destination before being transported to the designated transport destination.

[0026] (13) One aspect of the present invention is a computer program, wherein the computer is configured to function as a foreign object detection device, the foreign object detection device acquiring an image of the object to be detected by a camera device mounted on a conveyor for transporting scrap iron, and using the acquired image of the object to be detected to determine whether the scrap iron contains foreign objects.

[0027] Invention Effects

[0028] According to the present invention, even when a conveyor is used in the handling of scrap iron for recycling, it is possible to determine with high accuracy whether foreign matter has been mixed into the scrap iron. Attached Figure Description

[0029] Figure 1 This is a schematic block diagram showing the system structure of the determination system 100 according to the first embodiment of the present invention.

[0030] Figure 2 This is a diagram representing an example of the display output to the user.

[0031] Figure 3 This is a diagram illustrating an application example of the determination system 100.

[0032] Figure 4 This is a diagram showing a variation of the setup of the shooting device 10.

[0033] Figure 5 This is a diagram showing a variation of the setup of the shooting device 10.

[0034] Figure 6 This is a schematic block diagram illustrating a specific example of the functional structure of the foreign object detection device 20.

[0035] Figure 7 This is a diagram representing a specific example of an image of scrap metal used as teaching data.

[0036] Figure 8 This is a flowchart illustrating a specific example of the processing by the foreign object detection device 20.

[0037] Figure 9 This is a diagram illustrating a first specific example of the installation of the determination system 100 of the first embodiment.

[0038] Figure 10 This is a diagram illustrating a second specific example of the installation of the determination system 100 of the first embodiment.

[0039] Figure 11 This is a diagram illustrating a third specific example of the installation of the determination system 100 of the first embodiment.

[0040] Figure 12 This is a diagram illustrating a fourth specific example of the installation of the determination system 100 of the first embodiment.

[0041] Figure 13 This is a diagram illustrating a specific example of the state of heavy machinery 40.

[0042] Figure 14 This is a diagram illustrating a specific example of the state of heavy machinery 40.

[0043] Figure 15 This is a rough diagram representing region C.

[0044] Figure 16 This is a schematic block diagram illustrating the system structure of the determination system 100 according to the second embodiment of the present invention.

[0045] Figure 17 This is a schematic block diagram illustrating a specific example of the functional structure of the timing determination device 50.

[0046] Figure 18 This is a diagram showing a specific example of the teaching data used for learning processing of the timing decision model.

[0047] Figure 19 This is a flowchart illustrating a specific example of the processing of the determination system 100 in the second embodiment.

[0048] Figure 20 This is a diagram illustrating a first specific example of the installation of the determination system 100 of the second embodiment.

[0049] Figure 21 This is a diagram illustrating a second specific example of the installation of the determination system 100 of the second embodiment.

[0050] Figure 22 This is a diagram illustrating a third specific example of the installation of the determination system 100 of the second embodiment.

[0051] Figure 23 This is a diagram illustrating a fourth specific example of the installation of the determination system 100 of the second embodiment.

[0052] Figure 24 This is a diagram illustrating a fifth specific example of the installation of the determination system 100 of the second embodiment.

[0053] Figure 25 This is a schematic diagram showing an example of the hardware structure of the information processing apparatus 90 applied in this embodiment.

[0054] Figure 26 This is a flowchart illustrating a first variation of the processing of the determination system 100 in the second embodiment.

[0055] Figure 27 This is a flowchart illustrating a second variation of the processing of the determination system 100 in the second embodiment. Detailed Implementation

[0056] [First Implementation Method]

[0057] Figure 1 This is a schematic block diagram illustrating the system structure of the determination system 100 according to the first embodiment of the present invention. The determination system 100 is used to determine whether or not foreign objects contained in the scrap iron of the determination object are present, based on an image of the scrap iron of the determination object captured by the imaging device 10 (hereinafter referred to as the "determination object image"). In addition, in this embodiment, "imaging" refers to generating electronic image data based on light entering the imaging element of the imaging device 10. That is, "imaging" is not limited to recording image data on a non-volatile recording medium.

[0058] The determination system 100 includes an imaging device 10, a foreign object detection device 20, and an output device 30. The imaging device 10 and the foreign object detection device 20 are connected in such a way that data from an image captured by the imaging device 10 can be output to the foreign object detection device 20. The imaging device 10 and the foreign object detection device 20 can also be integrally configured and input / output image data via a bus. Alternatively, the imaging device 10 and the foreign object detection device 20 can be connected via a cable for image data input / output. The imaging device 10 and the foreign object detection device 20 can input / output image data via short-range wireless communication or via a network (e.g., LAN, mobile communication network, Internet). Furthermore, in this embodiment, "image acquisition" refers to the foreign object detection device acquiring image data generated by imaging.

[0059] The foreign object detection device 20 and the output device 30 are connected in such a way that they can output detection result data representing the detection result of the foreign object detection device 20 to the output device 30. The foreign object detection device 20 and the output device 30 may also be integrally configured and input / output the detection result data via a bus. Alternatively, the foreign object detection device 20 and the output device 30 may be connected via a cable for input / output of the detection result data. The foreign object detection device 20 and the output device 30 may also input / output the detection result data via short-range wireless communication or via a network (e.g., LAN, mobile communication network, Internet).

[0060] The imaging device 10 is a device equipped with at least an imaging element, configured to capture images of scrap metal, the object to be determined. The captured images can be still images or moving images. The imaging device 10 is configured using, for example, a digital still camera, a digital camera, a smartphone, a tablet computer, or a smart camera. The imaging device 10 outputs the captured image data (especially the image of the object to be determined) to the foreign object determination device 20. Furthermore, multiple imaging devices 10 can be installed in a single determination system 100. For example, multiple imaging devices 10 can be installed on a single piece of heavy machinery 40, as described later. More specifically, as described later... Figures 13-15 The example shows the location and orientation of multiple camera devices 10 positioned over heavy machinery 40. For example, Figure 13 The camera device 10 is shown in the indicated position and orientation. Figure 14 The camera device 10 is shown in the indicated position and orientation. Figure 15 The camera device 10, with the position and orientation shown, can also be installed on the same heavy machinery 40. Furthermore, the heavy machinery 40 is a specific example of a transport machine.

[0061] The foreign object detection device 20 determines whether the scrap iron contains a specified foreign object based on an image of the object to be detected (an image of the object containing scrap iron) captured by the imaging device 10. Specific examples of the specified foreign objects to be detected include objects such as (1) to (4). (1) Objects containing a large amount of elements (residual elements) that are difficult to remove in the steelmaking process. (2) Enclosed objects (e.g., gas cylinders, cans, etc.) that pose a risk of explosion in the steelmaking process (e.g., melting process). (3) Large or high-strength objects that pose a risk of damaging the crusher when fed into a crusher or similar shredder. (4) Combustible materials that could cause flames or other damage in the steelmaking process. The foreign object detection device 20 outputs data representing the detection result (detection result data) to the output device 30.

[0062] The output device 30 performs output processing based on the determination result of the foreign object detection device 20. The output device 30 is a device that outputs information in a manner that is understandable to the user (e.g., an operator using the detection system 100). The output device 30 can be configured as, for example, an image output device (e.g., a monitor) capable of displaying text and images, a device capable of outputting sound (e.g., a speaker, headphones), or an information device capable of connecting to or integrating these devices. Specific examples of such information devices include smartphones, tablets, and dedicated devices. Figure 2 This is a diagram illustrating an example of the display output to the user. A motor, acting as a foreign object, is detected within scrap metal held by a lifting magnet; a rectangle is generated around the motor's location. For example... Figure 2 As shown, the confidence level of the foreign object determination model for the detection results can also be displayed (in...). Figure 2 The mean value is 0.93. The minimum value is 0.0, and the maximum value is 1.0.

[0063] Figure 3 This is a diagram illustrating an application example of the determination system 100. For example... Figure 3 As shown, the determination system 100 can also be applied to mobile heavy machinery 40. Figure 3 In the example, heavy machinery 40 is a device capable of transporting scrap metal. Heavy machinery 40 can also be a device capable of moving scrap metal a longer distance by its own power. Heavy machinery 40 can also be, for example, a hydraulic excavator or similar device. Heavy machinery 40 is, for example, composed of a main body 403, a movable body 401 movable relative to the main body 403, a traveling body 406 that moves the heavy machinery 40, and a connecting device that connects the main body 403, etc., in a displacement manner, such as a slewing device that rotatably connects to the traveling body 406. Figure 3 In the example, a lifting magnet 402 is installed at the front end of the movable body 401 of the heavy machinery 40 as a conveying accessory (conveyor). The movable body 401 may include, for example, a boom 401a and a stick 401b, or movable parts other than the boom 401a and the stick 401b. The accessory installed at the front end of the movable body 401 is not limited to the lifting magnet 402. The accessory installed at the front end of the movable body 401 can be of any structure as long as it can hold the scrap metal in the air. For example, an accessory that can hold (grip) the scrap metal by opening and closing multiple movable parts, such as a grab bucket or alligator clamp, can also be used. The main body 403 includes, for example, an operator's cab 404 forming a space for the operator to sit in, and a rotating body 405 (displacement body) that rotates (displaces) together with the operator's cab 404.

[0064] Heavy machinery 40 lifts the scrap metal located in area A and moves it to area B. Heavy machinery 40 can operate according to the operator's commands or under the control of a pre-defined information processing device 90. Figure 3 In the example, the heavy machinery 40 is equipped with a camera 10, a foreign object detection device 20, and an output device 30. The camera 10 may be installed, for example, on the main body 403 of the heavy machinery 40 (e.g., in the control room 404 or rotating body 405 near the control room 404), facing a predetermined direction (e.g., towards the front end of the movable body 401). Specifically, the camera 10 may also be positioned facing forward and downward from the main body 403. Area A and Area B can each be areas of any shape. For example, one or both of Area A and Area B may be a flat surface such as the ground or floor, a cargo box of a truck or railway freight car, a ship's hold, a container for handling scrap metal (e.g., a pot-shaped container), an inlet for feeding scrap metal into a scrap metal handling device (e.g., a scrap chute), or other areas. As a specific example, Area A may be a cargo box, and Area B may be the ground designated as the scrap metal storage area.

[0065] Figure 4 and Figure 5 This is a diagram showing a variation of the arrangement of the imaging device 10. In the heavy machinery 40, the imaging device 10 can be arbitrarily positioned anywhere as long as it can capture images of the scrap metal that is the object of judgment. Figure 4 As shown, the camera device 10 can also be installed at a relatively low position within the movable body 401 (e.g., relatively close to the operator's cab 404). In this case, the camera device 10 can also be installed facing downwards at an angle. Figure 5 As shown, the imaging device 10 can also be installed at a relatively high position within the movable body 401 (e.g., close to a joint portion of the movable body 401). In this case, the imaging device 10 can also be installed facing downwards or directly downwards. Figure 5 In the manner shown, when the shooting device 10 is set in a downward-facing position, it can also be used with... Figure 4 Compared to setting the imaging device 10 at an angle close to directly downwards, the method shown places the imaging device 10 at a position higher than the control room 404 (movable body 401). Therefore, it is possible to photograph scrap metal over a wider area, thus preventing the possibility of only photographing a portion. As a result, the presence or absence of foreign objects can be determined with greater precision. Furthermore, by... Figure 4 and Figure 5This configuration allows for imaging from angles different from those in the control room 404. Therefore, the operator can visually confirm the presence of foreign objects from the control room 404, and determine the presence of foreign objects based on multiple images taken from angles outside the line of sight, thereby reducing the chance of missing foreign objects. Figure 4 or Figure 5 In this case, the shooting device 10 can be positioned towards the front end of the movable body 401, or it can be positioned towards the area (area A) where the scrap metal to be judged is piled up when the heavy machinery 40 lifts the scrap metal, or it can be positioned towards the area (area B) where the scrap metal to be judged is piled up when the heavy machinery 40 lowers the scrap metal. The shooting device 10 may also have a movable body for changing the shooting direction. When the heavy machinery 40 moves the scrap metal from area A to area B, the shooting device 10 can also follow the movement of the scrap metal and change its shooting direction accordingly. For example, the shooting device 10 may also operate in a manner that always follows the position of the front end of the movable body 401 (e.g., the lifting magnet 402) to take pictures. Such following action of the shooting device 10 can be performed automatically by a device not shown, or it can be performed by a person (e.g., an operator).

[0066] The output device 30 is used, for example, in the operator's cab 404. The output device 30 can also output information to the operator of the heavy machinery 40 operating from the operator's cab 404. The output device 30 can be fixedly installed in the operator's cab 404 or detachably installed in the operator's cab. If detachable, the output device 30 can also be brought in from outside the operator's cab 404 when the operator is operating the machinery. The output device 30 does not necessarily need to be installed inside the operator's cab 404 of the heavy machinery 40; for example, it can be held by a person located outside the heavy machinery 40 (an operator different from the operator of the heavy machinery 40).

[0067] Figure 6 This is a schematic block diagram illustrating a specific example of the functional structure of the foreign object detection device 20. The input / output unit 21 acquires image data of the object to be detected from the imaging device 10. The input / output unit 21 outputs the detection result data to the output device 30. The input / output unit 21 can be configured using an interface with a bus or cable, or it can be configured using a communication device. If the input / output unit 21 is a communication device, it can also communicate with other devices via a network under the control of the control unit 23. In this case, the communication device can be either a wireless communication device or a wired communication device.

[0068] Storage unit 22 stores data used by control unit 23. Storage unit 22 can also function as foreign object detection model storage unit 221, for example. Foreign object detection model storage unit 221 stores the foreign object detection model. The foreign object detection model is a function or logic used to determine the presence or absence of a specified foreign object in a determination object image, accompanied by specified input / output information.

[0069] The foreign object detection model is generated in advance through model building processing. Such model building processing can be performed by other devices, by this device (foreign object detection device 20), or by human operation. Hereinafter, the foreign object detection model stored in the foreign object detection model storage unit 221 will be described.

[0070] The foreign object detection model can be obtained, for example, through model building processing using known data. The foreign object detection model can be a learned model obtained through supervised learning processing using multiple known data points (teaching data) with correct labels, a model obtained through statistical processing using known data, or other models. Specific examples of learning processing include multivariate analysis, machine learning, and deep learning. As a learning process, a classification learning process can be used based on the correct labels used in the teaching data, or a regression learning process can be used. The control unit 23 determines whether there are foreign objects in the scrap iron to be detected using the foreign object detection model stored in the foreign object detection model storage unit 221. The teaching data in this embodiment will be described below.

[0071] As teaching data, each image of scrap metal in a state where it is lifted by an accessory mounted on the front end of the movable body of the heavy machinery can be used in combination with the correct label (indicating the presence or absence of foreign objects). For example, images captured under conditions similar to those of the imaging device 10 of the heavy machinery 40 used in the application determination system 100 can be used as teaching data. For instance, if the scrap metal captured by the imaging device 10 is in a state where it is piled up in area A or area B, images of the piled-up scrap metal can be used as teaching data. Similarly, if the scrap metal captured by the imaging device 10 is in a state where it is lifted by a lifting magnet, images of the scrap metal lifted by the lifting magnet can be used as teaching data. In this case, the lifting unit is not limited to the lifting magnet; images of scrap metal lifted by other accessories can also be used as teaching data.

[0072] As teaching data, images of scrap metal containing foreign objects and images of scrap metal without foreign objects can be used. Among the images of scrap metal containing foreign objects, images taken in a state where foreign objects are actually present can be used, or images taken in a state where foreign objects are actually absent can be used to synthesize images of foreign objects through image processing.

[0073] Figure 7 This is a diagram representing a specific example of an image of scrap metal used as teaching data. Figure 7 (A) and Figure 7 (B) are images obtained by photographing scrap iron in a state of being lifted by the lifting magnet 402. Figure 7 In (A), the scrap iron does not contain foreign objects. On the other hand, in Figure 7 In (B), scrap iron is included as a foreign object, specifically motor 81. This allows for the capture of images taken without any foreign object present (e.g.,...). Figure 7 (A) and images taken in a state containing foreign objects (e.g.) Figure 7 (B) is used as teaching data.

[0074] As a type of foreign object detection model, it can also be used Figure 7 (A) and Figure 7 (B) shows an image classification model that outputs whether an image contains foreign objects when given an input image. In this case, if a foreign object is present, it can also output what kind of foreign object it contains and its type. Furthermore, as other types of foreign object detection models, it can output not only... Figure 7 (A) and Figure 7 (B) shows that when an image is input, it can indicate whether the image contains foreign objects, and can also output information about where the foreign objects are located in the image if they are present. Methods for outputting the location of such foreign objects include object detection models that enclose foreign objects in the image with a bounding rectangle, and segmentation models that fill in the foreign objects in the image; either one can be used. The teaching data for the object detection model includes not only... Figure 7 (A) and Figure 7 An image like the one shown in (B) is for images containing foreign objects. Figure 7 (B) also includes information about the circumscribed rectangle surrounding the foreign object. The teaching data for the segmentation model not only includes... Figure 7 (A) and Figure 7 An image like the one shown in (B) is for images containing foreign objects. Figure 7 (B) also includes region information, which is the set of pixels in the image corresponding to the location of the foreign object.

[0075] The types of foreign object detection models are not limited to those generated through taught learning as described above. A known anomaly detection model can also be used: an image of scrap metal in a state without foreign objects is learned without teaching, generating a fully learned model that incorporates the features of such scrap metal. In practical applications, when an image of scrap metal containing foreign objects is input, the part containing the foreign object is detected based on the features not learned. Regarding known taught learning detection models and anomaly detection models, for example, the model described in International Publication No. 2022 / 260133 can also be used.

[0076] The processor of the control unit 23 executes the program, thereby functioning as the information control unit 231 and the foreign object detection unit 232.

[0077] The information control unit 231 controls the input and output of information. For example, the information control unit 231 obtains data (image data) of the image of the target object from the imaging device 10 via the input / output unit 21. For example, the information control unit 231 outputs data (judgment result data) representing the judgment result of the control unit 23 to the output device 30 via the input / output unit 21.

[0078] The foreign object detection unit 232 acquires the foreign object detection model stored in the foreign object detection model storage unit 221 and the image of the detection target acquired from the imaging device 10 to perform foreign object detection processing. Through foreign object detection processing, it determines whether the image of the detection target captured by the imaging device 10 contains a foreign object. That is, it determines whether the scrap iron, which is the detection target, contains a specified foreign object.

[0079] Figure 8 This is a flowchart illustrating a specific example of the processing of the foreign object detection device 20. First, the imaging device 10 takes an image (step S101), and the information control unit 231 outputs image data from the imaging device 10 via the input / output unit 21. The foreign object detection unit acquires this image data (step S102). The foreign object detection unit 232 performs a detection process using at least the foreign object detection model and the image data (step S103). The information control unit 231 sends information indicating the detection result to the output device 30 (step S104). Furthermore, this transmission may, for example, only occur when the foreign object is detected.

[0080] Figure 9 This is a diagram illustrating a first specific example of the installation of the determination system 100 according to the first embodiment. Figure 9 In the determination system 100 shown, the imaging device 10, the foreign object detection device 20, and the output device 30 constitute a determination device 200. This determination device 200 may also include an operating unit (touch panel, buttons, keyboard, etc.) for accepting operations from users such as operators. The user may, for example, be the operator of heavy machinery 40.

[0081] The judgment device 200 can also be operated by a user, and the imaging device 10 operates according to the operation, thereby capturing an image of the scrap metal to be judged. Alternatively, it can repeatedly capture images or capture moving images without relying on user operation. The image of the judgment object generated by the imaging device 10 is input to the foreign object judgment device 20, which determines whether a foreign object is present. The judgment result is output to the user by the output device 30. Such a judgment device 200 can also be constructed using, for example, a smartphone or tablet computer. In this case, by launching an application installed on the smartphone or tablet computer, the smartphone or tablet computer functions as the foreign object judgment device 20. Furthermore, the imaging device 10 and the output device 30 can be constructed using devices pre-installed on the smartphone or tablet computer, or they can be constructed using devices connected to the smartphone or tablet computer.

[0082] Figure 10 This is a diagram illustrating a second specific example of the installation of the determination system 100 according to the first embodiment. Figure 10 In the determination system 100 shown, the imaging device 10 and the foreign object detection device 20 constitute the determination device 200, and the output device 30 constitutes other devices. The determination device 200 configured in this way can also be set up... Figures 3-5 In this case, the determination device 200 and the output device 30 are connected in a manner that enables communication (e.g., wireless communication). Therefore, the determination device 200 and the output device 30 may also each be equipped with a communication device.

[0083] The determination device 200 can be configured using, for example, a smartphone or tablet, or it can be configured as a device that adds information processing functions to the shooting device, such as a smart camera. This information processing function can be integrated into a single enclosure with the shooting device 10, or it can be configured by connecting a single-board computer to the shooting device 10. The information processing function functions as the foreign object determination device 20. Additionally, the output device 30 can also be configured using a smartphone or tablet. In this case, the output device 30 can be configured using devices already present in the smartphone or tablet, or it can be configured using devices connected to the smartphone or tablet.

[0084] Figure 11 This is a diagram illustrating a third specific example of the installation of the determination system 100 of the first embodiment. In Figure 11 In the determination system 100 shown, the foreign object detection device 20 and the output device 30 constitute the determination device 200, and the imaging device 10 constitutes another device. The imaging device 10 configured in this way can, for example, be installed in... Figures 3-5In locations such as those where the camera device 10 is installed as part of heavy machinery 40, the camera device 10 and the determination device 200 can be connected communicatively (e.g., wirelessly). Therefore, the camera device 10 and the determination device 200 may also each be equipped with a communication device.

[0085] The determination device 200 may also be set, for example, in a location where... Figures 3-5 The determination device 200 is installed in a location such as the output device 30 of the heavy machinery 40. The determination device 200 receives an image of the object to be determined from the imaging device 10 via communication and performs determination processing. The determination device 200 outputs the determination result via the output device 30. The determination device 200 can also be configured using, for example, a smartphone or tablet computer. In this case, the output device 30 can be configured using devices already present in the smartphone or tablet computer, or it can be configured using a device connected to the smartphone or tablet computer.

[0086] Figure 12 This is a diagram illustrating a fourth specific example of the installation of the determination system 100 of the first embodiment. In Figure 12 In the determination system 100 shown, the imaging device 10 and the output device 30 constitute a terminal device 300, and the foreign object determination device 20 constitutes another device. The foreign object determination device 20 and the terminal device 300 can be communicatively connected via a communication path such as a network 70. The foreign object determination device 20 can also be configured using information processing devices such as server devices or cloud computing.

[0087] The terminal device 300 configured in this way can, for example, be set in... Figures 3-5 The terminal device 300, configured as an output device 30 for heavy machinery 40, may also include an operating unit (touch panel, buttons, keyboard, etc.) for accepting operations from users such as operators. The user may be, for example, the operator of the heavy machinery 40. The terminal device 300 can also be operated by the user, using the imaging device 10 to capture images of the scrap metal to be identified, based on the user's actions. Alternatively, it can repeatedly capture images or capture moving images without relying on user operation. The image of the identified object generated by the imaging device 10 is transmitted to the foreign object identification device 20 via communication through a communication device.

[0088] When the foreign object detection device 20 receives an image of the object to be detected, it determines whether the received image contains a foreign object. The foreign object detection device 20 then sends information indicating the determination result to the terminal device 300. When the terminal device 300 receives the determination result from the foreign object detection device 20, the output device 30 outputs the determination result to the user. Such a terminal device 300 can be configured, for example, using a smartphone or tablet computer. Furthermore, the imaging device 10 and the output device 30 can be configured using devices pre-existing in smartphones or tablet computers, or they can be configured using devices connected to smartphones or tablet computers.

[0089] The above uses Figures 9-12 A specific example of the installation of the determination system 100 according to the first embodiment has been described, but the installation method of the determination system 100 is not limited to the specific example described above. For example, such as Figure 1 As shown, the imaging device 10, the foreign object detection device 20, and the output device 30 can also be installed as different devices. In this case, for example, the imaging device 10 and the output device 30 are respectively as follows: Figures 3-5 As shown, the foreign object detection device 20 can also be configured using a server device, cloud, or other information processing device 90. In this case, the imaging device 10, the foreign object detection device 20, and the output device 30 can also each be equipped with a communication device.

[0090] An example of the operation of the determination system 100 configured in this way will be explained. For example, the imaging device 10 may be configured to repeatedly perform imaging at predetermined time intervals, and send the images of the determination object obtained by imaging to the foreign object determination device 20 according to certain rules (successively or at certain intervals). In this case, the operator of the heavy machinery 40 may move the scrap metal from area A to area B without being specifically aware of the imaging. If the output device 30 outputs information indicating the presence of foreign objects, the operator of the heavy machinery 40 may also perform the operation of removing foreign objects from the scrap metal of the determination object.

[0091] The operator of the heavy machinery 40 moves the heavy machinery 40 to approach area A. After moving the heavy machinery 40 to a position where it is so close to area A that the imaging device 10 is facing area A, the operator operates the imaging device 10. The operation of the imaging device 10 can be performed, for example, by directly touching buttons or a touch panel provided on the imaging device 10, or by using an operating device such as a remote control connected to the imaging device 10 via short-range wireless communication, or by the operator issuing specific statements (e.g., "shoot") through voice recognition. By performing such operations, images of the scrap metal piled up in area A can be captured. The imaging device 10 can also capture images of the scrap metal in area A without particularly relying on the operator's operation, for example, by repeatedly taking pictures or capturing moving images.

[0092] The operator of the heavy machinery 40 uses the heavy machinery 40 (especially the movable body 401) to lift the scrap metal. When operating the shooting device 10, the operator of the heavy machinery 40 can also take pictures of the lifted scrap metal by operating the shooting device 10 in this state. Figure 13 This is a diagram illustrating a specific example of the state of heavy machinery 40. In Figure 13 In this method, the imaging device 10 takes pictures with the lens facing upwards or directly upwards, so that the imaging range includes the lifting magnet 402 when the movable body 401 is raised. The imaging device 10 can be fixed in this orientation, or it can be configured to have a movable body that allows the orientation of the lens to be changed. By taking pictures in this state, the imaging device 10 can capture images of the raised scrap metal viewed from below. Especially when using the lifting magnet 402, since scrap metal is attached below, more images of scrap metal can be captured. By capturing such images, the presence or absence of foreign objects can be determined with higher precision. In addition, when the imaging device 10 is positioned towards the front end of the movable body 401, images of the raised scrap metal can be captured without relying particularly on the operator's operation, for example, by repeatedly taking pictures or capturing dynamic images. Furthermore, in Figure 13 When the imaging device 10 is performing imaging under such conditions, the scrap metal is being lifted. Therefore, the possibility of the scrap metal falling from the front end of the movable body 401 and colliding with the lens or other components of the imaging device 10 is relatively high. Furthermore, this makes the movable body 401 of the heavy machinery... Figure 13 That state also requires time, and productivity may decrease. Even with the possibility of such a collision and the possibility of reduced productivity, the presence or absence of foreign objects can be determined with higher precision as described above, so such a structure can also be adopted in this embodiment.

[0093] The operator of the heavy machinery 40 uses the heavy machinery 40 (especially the movable body 401) to lift scrap iron. The operator of the heavy machinery 40 can also manipulate the heavy machinery (especially the movable body 401) so that the lifted scrap iron is photographed by the photographing device 10 over a larger area. For example, it is also possible to manipulate the holding part of the scrap iron in the heavy machinery 40 that is lifting the scrap iron (e.g., the front end of the movable body 401) toward the photographing device. As a more specific example of such manipulation, there is manipulation to make the bottom surface (the surface on which the scrap iron is attached) of the front end of the movable body 401 (e.g., the lifting magnet 402) face the photographing device 10. Figure 14 This diagram illustrates a specific example of the state of heavy machinery 40. By taking pictures in this state using the imaging device 10, it is possible to capture more images of the lifted scrap metal. Figure 14 In this process, the imaging device 10 takes pictures with the lens facing horizontally, downwards, or upwards, so that the lifting magnet 402 is included in the shooting range. The imaging device 10 can be fixed in this orientation, or it can be configured as a device with a movable body that can change the orientation of the lens. By taking such pictures, the presence or absence of foreign objects can be determined with higher precision. When the imaging device 10 is facing the same direction as the operator, the operator can also check for foreign objects with their own eyes by performing such operation, and a dual inspection can be performed by the judgment system and visual inspection. When the operator operates the imaging device 10 to take pictures, the bottom surface of the lifting magnet 402 can be photographed with the imaging device 10 facing the imaging device 10 by operating the imaging device 10 in this state. In addition, when the imaging device 10 is set facing the front end of the movable body 401, the image of the scrap iron in the above state can be photographed without particularly relying on the operator's operation, for example, by repeatedly taking pictures or taking dynamic images.

[0094] The operator of the heavy machinery 40 can also manipulate the heavy machinery 40 (especially the movable body 401) to lower the scrap metal that was lifted once from area A to an area (area C) that is different from area A and area B. Figure 15 This is a rough diagram representing region C. Figure 15 In this system, the imaging device 10 captures images with the lens facing horizontally or diagonally downwards, encompassing the area C within its imaging range. The imaging device 10 can be fixed in this orientation or configured to have a movable body capable of changing the lens's orientation. When the operator of the heavy machinery 40 lowers the scrap metal into the area C, they can manipulate the heavy machinery (especially the movable body 401) to disperse the individual scrap metal components over a wide area with minimal overlap. This manipulation allows for clearer images of each scrap metal component, revealing the target object. For example… Figure 13 , Figure 14 As shown in the embodiment, when photographing a mass of scrap metal held at the front end of the movable body 401, it is difficult to detect foreign objects hidden inside the mass. To address this problem, in this embodiment, by temporarily dispersing the scrap metal parts in area C, such omissions in detection can be eliminated. Therefore, the determination of the presence or absence of foreign objects can be performed with higher accuracy. In addition, when the photographing device 10 and the operator are facing the same direction, by performing such operation, the operator can also visually check for the presence or absence of foreign objects, enabling a dual inspection through the determination system and visual inspection. When the operator operates the photographing device 10, the photographing device 10 can also be operated while operating the heavy machinery 40, thereby capturing images of the dispersed state of the individual scrap metal parts. Furthermore, it is also possible to capture images of the scrap metal in the above-described state without relying on the operator's operation, for example, by repeatedly taking pictures or capturing moving images. When the scrap metal is temporarily placed in area C for photographing, after the operator has photographed in area C, the scrap metal placed in area C is collected and moved to area B.

[0095] The features of the determination system 100 of the first embodiment, configured as described below, will be explained. Conventionally, there are systems where the movement path of the conveying device is fixed (hereinafter referred to as a "path-fixed system"), as described above. It is sufficient to photograph scrap iron at a certain point along this path, so the photographing device is fixedly installed. Therefore, in systems where the transport path of scrap iron is not fixed, such as when scrap iron is transported using heavy machinery 40, it is difficult to accurately determine the presence or absence of foreign objects. In contrast, in the determination system 100 of the first embodiment, images are obtained by photographing scrap iron placed in the transport source area A, the transport destination area B, or during the transport process of the heavy machinery 40 using a photographing device installed on the heavy machinery 40, and the presence or absence of foreign objects in the scrap iron is determined. The transport process refers to, for example, using a conveying device attached to the heavy machinery 40 to hold scrap iron in a certain area (e.g., area A), transporting it from that area to other areas (e.g., area B, area C), and releasing the holding at the transport destination area. The imaging device 10 moves along with the heavy machinery 40, thus allowing it to always be positioned close to the scrap metal, unlike a fixed imaging device 10. This enables high-precision determination of whether foreign objects are mixed into the scrap metal. Furthermore, the movement of the imaging device 10 with the heavy machinery 40 ensures that the distance between the imaging device 10 and the scrap metal remains approximately the same. This results in several advantages. For example, when a learned model obtained through learning processing of the teaching images is used to determine whether foreign objects are present, images can be taken from approximately the same distance and with approximately the same size as the foreign objects captured in the images used in the teaching images. Therefore, the presence of foreign objects mixed into the scrap metal of the recycling object can be determined with high precision. This effect is not limited to the case of using a learned model; it is also achieved when using techniques such as pattern matching. Additionally, since the distance remains approximately the same, by setting the camera focus to be on the scrap metal, images can always be taken with good focus. Furthermore, the operator's cab 404 of the heavy machinery 40 is mostly located relatively close to the scrap metal compared to the operator's cab of a path-fixing system such as a bridge crane. In path-fixing systems, the operator's cab is usually positioned above, requiring the operator to view the scrap metal from above. In this case, the scrap metal holding devices are often within the operator's field of vision, making it difficult to visually inspect the scrap metal. Therefore, using heavy machinery 40 to handle scrap metal, instead of a path-fixing system, has the advantage of allowing for closer visual inspection of the scrap metal. Additionally, the implementation cost of heavy machinery is generally much lower and faster than that of a path-fixing system. Therefore, path-fixing systems are less commonly implemented among scrap metal processing businesses.Therefore, the technology for using heavy machinery to process scrap iron, as in this embodiment, also has the following advantages: it enables more technicians to improve the accuracy of foreign object detection mixed into the scrap iron.

[0096] [Second Implementation]

[0097] Figure 16 This is a schematic block diagram illustrating the system structure of the determination system 100 according to the second embodiment of the present invention. The determination system 100 of the second embodiment differs from the determination system 100 of the first embodiment in that it also includes a timing determination device 50. The timing determination device 50 determines the timing of acquiring the image of the object to be determined used by the foreign object determination device 20 in the determination process. Therefore, the first embodiment is an embodiment where the user determines the timing of acquiring the image of the object to be determined and where the image of the object to be determined is acquired according to a certain rule (e.g., a certain interval), but the second embodiment is an embodiment where the determination system 100 sets the timing of acquiring the image of the object to be determined to be variable (free). Hereinafter, the timing determination device 50 will be described in detail.

[0098] Figure 17 This is a schematic block diagram illustrating a specific example of the functional structure of the timing determination device 50. The input / output unit 51 acquires image data of the object to be determined from the imaging device 10. The input / output unit 51 inputs data for timing determination processing from other devices. For example, the input / output unit 51 inputs information related to the control of the heavy machinery 40 (hereinafter referred to as "control information") from the heavy machinery 40 on which the imaging device 10 is installed. Control information may include, for example, information indicating the operator's actions, electrical signals from electrical equipment installed on the heavy machinery 40, control signals, or hydraulic signals from hydraulic equipment installed on the heavy machinery 40. The input / output unit 51 outputs information indicating the timing determination result to other devices (e.g., the imaging device 10 or the foreign object detection device 20). The input / output unit 51 may also output the timing image data shown by the determination result to the foreign object detection device 20. The input / output unit 51 can be configured using an interface with a bus or cable, or it can be configured using a communication device. When the input / output unit 51 is a communication device, it can also communicate data with other devices via a network under the control of the control unit 53. In this case, the communication device can be either a device for wireless communication or a device for wired communication.

[0099] Storage unit 52 stores data used by control unit 53. Storage unit 52 may also function as timing determination model storage unit 521, for example. Timing determination model storage unit 521 stores timing determination model. Timing determination model is a function or logic that uses predetermined input / output information to determine the timing of obtaining the image of the object to be determined used by foreign object determination device 20 in determination processing.

[0100] The timing determination model is generated in advance through model building processing. Such model building processing can be performed by other devices, by this device (timing determination device 50), or by human operation.

[0101] The processor of the control unit 53 executes the program, thereby functioning as the information control unit 531 and the timing determination unit 532.

[0102] The information control unit 531 controls the input and output of information. For example, the information control unit 531 obtains image data (image data) of the object to be determined from the imaging device 10 via the input / output unit 51. For example, the information control unit 531 obtains control information from the heavy machinery 40. For example, the information control unit 531 outputs data representing the determination result of the control unit 53 to the imaging device 10 and the foreign object detection device 20 via the input / output unit 51.

[0103] The timing determination unit 532 uses the timing determination model stored in the timing determination model storage unit 521, the image of the determination object obtained from the imaging device 10, and the control information obtained from the heavy machinery 40 to perform timing determination processing. By performing such timing determination processing, the timing determination unit 532 performs real-time determination processing on the image captured by the imaging device 10 and the control information obtained from the heavy machinery 40, and outputs the determination result appropriately via the input / output unit 51.

[0104] The timing determination model stored in the timing determination model storage unit 521 will be described below.

[0105] The timing determination model can also take one or more control information from the heavy machinery 40 as input, and determine whether the foreign object determination device 20 has obtained the timing of the image of the object to be determined for determination processing based on whether these control information meet preset conditions. Here, the preset conditions can be, for example, whether a control information is consistent with a certain value, whether it is above (below) a certain value, whether it exceeds (below) a certain value, or whether it enters a certain range. Multiple such conditions can also be set in multiple control information, and the conditions can be set by combining them into a logical expression.

[0106] Here are a few specific examples of such control information.

[0107] As a first example, for example... Figure 13 As shown, there is control information indicating that the operation of lifting scrap metal using the movable body 401 has been performed. This control information can also be information indicating the operation performed by the operator of the heavy machinery 40 (e.g., an electrical signal indicating the operation of a control lever). For example, the time period during which the operation information (electrical signal) indicating that the operator of the heavy machinery 40 is performing the operation of lifting scrap metal using the movable body 401 is obtained can be determined as the timing for obtaining the judgment object image. This control information can also be a hydraulic signal in the heavy machinery 40. Alternatively, a hydraulic signal is obtained to hydraulically move the movable body 401 so that the front end of the movable body 401 moves upward; after the action corresponding to this hydraulic signal is completed, the time period until a new hydraulic signal for moving the movable body 401 is obtained is determined as the timing for obtaining the judgment object image. Furthermore, multiple electrical signals and hydraulic signals can be combined to determine the timing for obtaining the judgment object image for judgment processing.

[0108] As a second example, for example... Figure 14 As shown, there is control information indicating that the bottom surface (the surface of the scrap metal) of the front end of the movable body 401 (e.g., the lifting magnet 402) has been manipulated towards the imaging device 10. Such control information can also be information indicating the manipulation performed by the operator of the heavy machinery 40 (e.g., an electrical signal indicating manipulation of a joystick). For example, the time period during which manipulation information (electrical signals) indicating that the operator of the heavy machinery 40 is manipulating the front end of the movable body 401 towards the imaging device 10 is obtained can be determined as the timing for acquiring the image of the judgment target. Such control information can also be a hydraulic signal in the heavy machinery 40. The time period during which the timing for acquiring the image of the judgment target is determined is the period during which a hydraulic signal is obtained for the movable body 401 to move hydraulically when the front end of the movable body 401 is facing the imaging device 10, and until a hydraulic signal is obtained for the movable body 401 to move again after the action corresponding to the hydraulic signal is completed. Alternatively, multiple electrical signals and hydraulic signals can be combined to determine the timing for acquiring the image of the judgment target for judgment processing.

[0109] As a third example, and as other specific examples of control information, such as... Figure 15As shown, there is control information indicating that an operation was performed to distribute the scrap metal components to area C in a manner that prevents them from piling up. Such control information could be, for example, control information such as: when the situation requiring the scrap metal to be lowered once from area A to area C is regularized, it indicates that an operation was performed to temporarily lift the scrap metal and then lower it. More specifically, as described below. Such control information could also be information indicating the content of the operation performed by the operator of the heavy machinery 40 (e.g., an electrical signal indicating the operation of a control lever). For example, it could be that, upon receiving operation information (electrical signal) indicating that the operator of the heavy machinery 40 performed an operation to hold the scrap metal at the front end of the movable body 401 (e.g., an operation to generate magnetic force by lifting magnet 402), and then the operator of the heavy machinery 40 performed an operation to release the holding of the scrap metal (e.g., an operation to stop the generation of magnetic force by lifting magnet 402), the time period until the next operation to hold the scrap metal is performed is determined to be the timing for acquiring the image of the determination object. Such control information could also be hydraulic signals in the heavy machinery 40. Alternatively, if a hydraulic signal is received to hold scrap metal at the front end of the movable body 401 (e.g., a hydraulic signal to close the accessory holding the scrap metal), and then a hydraulic signal is received to release the grip on the scrap metal (e.g., a hydraulic signal to open the accessory), the time period until the hydraulic signal to begin the next gripping of the scrap metal is determined to be the timing for acquiring the image of the judgment target. Alternatively, multiple electrical signals and hydraulic signals can be combined to determine the timing for acquiring the image of the judgment target for judgment processing.

[0110] The timing determination model can, for example, take the image captured by the imaging device 10 as input and output a determination result indicating whether the timing is within a specified range. One such timing determination model is one that determines whether the image captured by the imaging device 10 was taken from below while the scrap metal is being lifted by heavy machinery 40. Such a model can also be obtained by performing learning processing that uses images captured in such a state as teaching data. Figure 18 (A) is a diagram showing a specific example of the teaching data used to perform such learning processing. Figure 18In (A), the image is taken from below while the scrap iron is held by the lifting magnet 402 and the movable body 401 connected to the lifting magnet 402 is moved upward. By using images of the scrap iron being lifted by the heavy machinery 40 and viewed from below as teaching data for learning processing, a learned model capable of determining such a state based on the image can be obtained. This learned model can be used as a timing determination model. Alternatively, more simply, for images taken while the scrap iron is being lifted by the heavy machinery 40 and viewed from below, feature quantities representing a predetermined pattern are pre-calculated. Based on this, the same feature quantities are calculated for new images newly captured by the imaging device 10 and input into the timing determination unit 532, and compared with the pre-calculated image feature quantities to determine whether the new image was taken while the scrap iron is being lifted by the heavy machinery 40 and viewed from below.

[0111] As another specific example of a timing determination model, there is a model that determines whether an image captured by the imaging device 10 was captured with the bottom surface (the surface on which scrap metal adheres) of an accessory (e.g., a lifting magnet 402) mounted on the front end of the movable body 401 facing the imaging device 10. Such a model can also be obtained, for example, by performing learning processing that uses images captured in such a state as teaching data, as a learned model. Figure 18 (B) is a diagram showing a specific example of the teaching data used to perform such learning processing. Figure 18 In (B), the image is taken with the scrap iron held by the lifting magnet 402 and the bottom surface of the lifting magnet 402 facing the imaging device 10. By using images of the actual state where the bottom surface of the lifting magnet 402 faces the imaging device 10 as teaching data for learning processing, a learned model capable of determining such a state based on the image can be obtained. This learned model can be used as a timing determination model. Alternatively, more simply, for images taken with the bottom surface of the accessory facing the imaging device 10, feature quantities representing a predetermined pattern can be pre-calculated. Then, for new images newly captured by the imaging device 10 and input into the timing determination unit 532, the same feature quantities are calculated and compared with the pre-calculated image feature quantities to determine whether the new image was taken with the bottom surface of the accessory facing the imaging device 10.

[0112] As another specific example of a timing determination model, there is a model that determines whether an image captured by the imaging device 10 was taken in a state where the scrap metal parts are dispersed in area C in a manner that does not overlap. In other words, this timing determination model determines whether the scrap metal lifted at least once by the heavy machinery 40 was photographed in a state where it was placed in an area (e.g., area C) different from the transport destination before being transported to a designated transport destination (e.g., area B). Such a model can also be obtained, for example, by performing learning processing that uses images taken in such a state as teaching data, as a learned model. Figure 18 (C) is a diagram showing a specific example of the teaching data used to perform such learning processing. Figure 18 In (C), the scrap iron is placed in a dispersed state on the ground, and the image is obtained by photographing this state. By using the image of the actual state in which the scrap iron is dispersed on the ground as teaching data for learning processing, a learned model that can determine such a state based on the image can be obtained. Such a learned model can be used as a timing determination model. Alternatively, for an image of the state in which the scrap iron is dispersed on the ground, based on the pre-calculated feature quantities representing a predetermined pattern, the same feature quantities can be calculated for a new image newly photographed by the photographing device 10 and input into the timing determination unit 532, and compared with the pre-calculated image feature quantities to determine whether the new image is an image of the state in which the scrap iron is dispersed on the ground. Thus, the above description describes a method of mechanically using a timing determination model to determine the timing of the determination based on the state recognition of electrical signals, hydraulic signals, or acquired images.

[0113] Figure 19 This is a flowchart illustrating a specific example of the processing of the determination system 100 in the second embodiment. The imaging device 10 continuously captures images (step S201). The timing determination device 50 determines the image acquisition timing (hereinafter also referred to as "processing timing") for acquiring the image of the determination object used in the determination processing (step S202). The foreign object determination device 20 stands by until it is determined that the image acquisition timing is correct. If the image acquisition timing is correct (step S202: "Yes"), the foreign object determination device 20 acquires the image data captured at or immediately before that timing (step S203). Then, the acquired image data and the foreign object determination model are used for determination processing (step S204). Then, the foreign object determination device 20 sends information indicating the determination result to the output device 30 (step S205).

[0114] Next, use Figures 20-24 This section describes a specific example of the installation of the determination device 200.

[0115] in addition, Figures 20-24The determination device 200, image acquisition device 400, terminal device 300, and devices connected to these devices (capturing device 10, output device 30, image acquisition device 400) and terminal device 300 can also be configured using, for example, a smartphone or tablet computer. In this case, by launching an application installed on the smartphone or tablet computer, the smartphone or tablet computer functions as the determination device 200, etc. Furthermore, the devices connected to the determination device 200, etc., can also be configured using devices connected to smartphones or tablet computers.

[0116] Figure 20 This is a diagram illustrating a first specific example of the installation of the determination system 100 according to the second embodiment. Figure 20 In the determination system 100 shown, the imaging device 10, the foreign object detection device 20, the output device 30, and the timing determination device 50 constitute a determination device 200. This determination device 200 may also include an operating unit (touch panel, buttons, keyboard, etc.) for accepting operations from users such as operators. The user may, for example, be the operator of heavy machinery 40.

[0117] The imaging device 10 of the determination device 200 captures images at predetermined intervals without relying on user operation. The imaging device 10 can also continuously capture moving images. The timing determination device 50 determines the timing for acquiring the image of the object to be determined based on input data (images captured by the imaging device 10 and control information of the heavy machinery 40). When the foreign object determination device 20 determines that it is the timing for acquiring the image of the object to be determined for determination processing, the timing determination device 50 outputs the image (object to be determined) at that timing to the foreign object determination device 20. The foreign object determination device 20 determines the presence or absence of a foreign object by using the image (object to be determined) captured at that timing. The determination result is output to the user by the output device 30.

[0118] Alternatively, in the first specific example, the determination device 200 can also be operated by a user, and the imaging device 10 can be used to capture an image of the scrap metal to be determined by the user's operation. In this case, the foreign object determination device 20 also processes the object based on the determination result of the timing determination device 50.

[0119] Figure 21 This is a diagram illustrating a second specific example of the installation of the determination system 100 according to the second embodiment. Figure 21 In the determination system 100 shown, the imaging device 10, the foreign object detection device 20, and the timing determination device 50 constitute the determination device 200, and the output device 30 constitutes other devices. In this configuration, the determination device 200 and the output device 30 are connected in a manner that enables communication (e.g., wireless communication). Therefore, the determination device 200 and the output device 30 may also each be equipped with a communication device.

[0120] The determination device 200 can be configured as, for example, a smartphone or tablet, or as a device that adds information processing functionality to the shooting device, such as a smart camera. This information processing functionality can be integrated into the shooting device 10 within a single enclosure, or it can be configured by connecting a single-board computer to the shooting device 10. The information processing functionality functions as both the foreign object determination device 20 and the timing determination device 50.

[0121] The timing determination device 50 determines the timing for acquiring the image of the object to be determined based on the input data (images captured by the imaging device 10 and control information of the heavy machinery 40). When the foreign object determination device 20 determines that it is the right time to acquire the image of the object to be determined for determination processing, the timing determination device 50 outputs the image (object to be determined) at that timing to the foreign object determination device 20. The foreign object determination device 20 acquires the image (object to be determined) captured at that timing to determine whether a foreign object is present. The determination result is output to the user by the output device 30.

[0122] Figure 22 This is a diagram illustrating a third specific example of the installation of the determination system 100 according to the second embodiment. Figure 22 In the determination system 100 shown, the imaging device 10 and the timing determination device 50 constitute an image acquisition device 400, and the foreign object determination device 20 and the output device 30 constitute a determination device 200. The image acquisition device 400 configured in this way can also be, for example, installed in... Figures 3-5 In locations such as those where the imaging device 10 is installed as a camera on heavy machinery 40, the image acquisition device 400 and the determination device 200 are connected in a manner that enables communication (e.g., wireless communication). Therefore, the image acquisition device 400 and the determination device 200 may also each be equipped with a communication device.

[0123] The image acquisition device 400 sends the image captured by the imaging device 10 to the determination device 200 at a timing determined by the timing determination device 50. The determination device 200 may also be configured, for example, to... Figures 3-5 The determination device 200 is installed in a location such as the output device 30 of the heavy machinery 40. The determination device 200 receives an image of the object to be determined from the image acquisition device 400 via communication and performs determination processing. The determination device 200 outputs the determination result via the output device 30.

[0124] Figure 23 This is a diagram illustrating a fourth specific example of the installation of the determination system 100 according to the second embodiment. Figure 23 In the determination system 100 shown, the imaging device 10 and the timing determination device 50 constitute an image acquisition device 400, and a foreign object determination device 20 and an output device 30 are respectively provided. This image acquisition device 400 can also be configured in, for example, as... Figures 3-5 In locations such as those where the imaging device 10 is installed as part of heavy machinery 40, the image acquisition device 400 and the foreign object detection device 20 are connected in a manner capable of communication (e.g., wireless communication). Therefore, both the image acquisition device 400 and the foreign object detection device 20 may also be equipped with communication devices. The foreign object detection device 20 may, for example, be configured using an information processing device 90 such as a server device or a cloud computing device.

[0125] The image acquisition device 400 sends the image captured by the imaging device 10 to the foreign object detection device 20 at a timing determined by the timing determination device 50. The foreign object detection device 20 may also be located in... Figures 3-5 In locations such as those where the output device 30 of heavy machinery 40 is installed, the foreign object detection device 20 receives an image of the object to be detected from the image acquisition device 400 via communication and performs detection processing. The foreign object detection device 20 outputs the detection result through the output device 30.

[0126] Figure 24 This is a diagram illustrating a fifth specific example of the installation of the determination system 100 according to the second embodiment. Figure 24 In the determination system 100 shown, the imaging device 10, the timing determination device 50, and the output device 30 constitute a terminal device 300, while the foreign object determination device 20 constitutes another device. The foreign object determination device 20 and the terminal device 300 can be communicatively connected via a network or other communication path. The foreign object determination device 20 can also be configured using, for example, a server device, a cloud, or other information processing device 90.

[0127] The terminal device 300 configured in this way can, for example, be set in... Figures 3-5 The terminal device 300 is installed in a location such as the output device 30 of the heavy machinery 40. The imaging device 10 of the terminal device 300 takes pictures at predetermined time intervals without relying on user operation. The imaging device 10 can also continuously capture moving images. The timing determination device 50 determines the timing based on the input data (images captured by the imaging device 10 and control information of the heavy machinery 40). When the foreign object detection device 20 determines that it is time to obtain an image of the object to be detected for detection processing, the terminal device 300 sends image data to the foreign object detection device 20.

[0128] When the foreign object detection device 20 receives image data from the terminal device 300, it uses the image to determine whether there is a foreign object. The determination result is output to the user by the output device 30. Such a detection device 200 can also be configured using, for example, a smartphone or tablet computer.

[0129] The above uses Figures 20-24A specific example of the installation of the determination system 100 in the second embodiment has been described, but the installation method of the determination system 100 is not limited to the specific example described above. For example, such as Figure 16 As shown, the imaging device 10, the foreign object detection device 20, the output device 30, and the timing detection device 50 can also be installed as different devices. In this case, for example, the imaging device 10 and the output device 30 are respectively as follows: Figures 3-5 As shown, the foreign object detection device 20 can also be configured using a server device, cloud, or other information processing device 90. The timing detection device 50 can also be configured to connect to the imaging device 10 without a network. In this case, the imaging device 10, the foreign object detection device 20, the output device 30, and the timing detection device 50 can each also have a communication device.

[0130] In the determination system 100 of the second embodiment configured in this way, the timing determination device 50 determines the timing at which a suitable image is likely to be captured. Then, the image captured at the timing of the determination result is acquired for foreign object determination. Therefore, it is possible to determine with high accuracy whether foreign objects are mixed in with the scrap iron of the recycling object. In addition, only the image captured at the timing of the determination result is sent to the foreign object determination device 20 via the network. Therefore, the resources and communication data required to send the image of the object to be determined can be reduced. In addition, by determining the timing and narrowing the processing to that timing, power consumption can be suppressed to a low level. In particular, in environments where the power capacity is limited, such as the power supply installed in heavy machinery, by suppressing power consumption, the imaging device, the foreign object determination device, and the output device can all operate through the wiring / power supply from the heavy machinery, thereby achieving the advantages of a compact device, or being able to install multiple cameras as a result of suppressing the power consumption of each camera. In addition, by determining the timing and narrowing the processing to that timing, over-detection of operator output can be suppressed. That is, in cases where the foreign object detection device 20 falsely detects scrap iron even though it is clear that no such scrap iron was actually captured, it can suppress such false detections from being output to the operator.

[0131] Figure 25 This is a schematic diagram illustrating an example of the hardware structure of the information processing apparatus 90 applied in this embodiment. The information processing apparatus 90 includes a processor 91, a main storage device 92, a communication interface 93, an auxiliary storage device 94, an input / output interface 95, and an internal bus 96. The processor 91, the main storage device 92, the communication interface 93, the auxiliary storage device 94, and the input / output interface 95 are connected to each other via the internal bus 96 in a manner that enables them to communicate with each other.

[0132] Specific examples of such an information processing device 90 include smartphones, tablets, personal computers, server devices, PLCs (Programmable Logic Controllers), and dedicated devices. The processor 91 and main storage device 92 can also be constructed using general-purpose devices such as CPUs (Central Processing Units). Alternatively, they can be constructed using ASICs (Application Specific Integrated Circuits), PLDs (Programmable Logic Devices), or FPGAs (Field Programmable Gate Arrays). The program executed by the processor 91 can also be recorded on a computer-readable recording medium. Computer-readable recording media include removable media such as floppy disks, optical disks, ROMs, CD-ROMs, semiconductor storage devices (e.g., SSDs), hard disks built into the computer system, and other storage devices. The program can also be transmitted via electrical communication lines. The communication interface 93 is constructed using a communication device. The auxiliary storage device 94 is constructed using storage devices such as magnetic hard disks or semiconductor storage devices. Input / output interface 95 is the interface between external devices and information processing device 90.

[0133] The information processing device 90 can also be applied to, for example, the foreign object detection device 20 and the timing detection device 50. In this case, the input / output unit 21 and the input / output unit 51 can be configured using either a communication interface 93 or an input / output interface 95. Similarly, the storage unit 22 and the storage unit 52 can be configured using an auxiliary storage device 94. Furthermore, the control unit 23 and the control unit 53 can be configured using a processor 91 and a main storage device 92. Additionally, the information processing device 90 can also be applied to the detection device 200 and the image acquisition device 400.

[0134] (Modified example)

[0135] Figure 26This is a flowchart illustrating a first variation of the processing of the determination system 100 in the second embodiment. First, the timing determination device 50 uses one or more control information from the heavy machinery 40 to determine the image capture timing (also called the processing timing) (step S301). Until the capture timing is determined, the other devices of the determination system 100 remain in standby mode. If the capture timing is determined (step S301: "Yes"), the imaging device 10 captures an image at that timing (step S302). The foreign object detection device 20 acquires the captured image data (step S303). Then, the acquired image data and the foreign object detection model are used for determination processing (step S304). Then, the foreign object detection device 20 sends information indicating the determination result to the output device 30 (step S305). In this way, by having all steps of image capture, image acquisition, foreign object detection processing, and result output performed only during the capture timing, power consumption can be suppressed.

[0136] Figure 27 This is a flowchart illustrating a second variation of the processing of the determination system 100 in the second embodiment. The imaging device 10 continuously captures images (step S401). The foreign object determination device 20 continuously acquires image data captured by the imaging device 10 (step S402). Then, determination processing is performed using the acquired image data and the foreign object determination model (step S403). The timing determination device 50 uses one or more control information from the heavy machinery 40, or the captured image data, to determine the output timing (also called processing timing) of the determination result of the output foreign object determination model (step S404). Before determining that it is the output timing, the foreign object determination device 20 waits for the output of the determination result. If it is determined that it is the output timing (step S404: "Yes"), the foreign object determination device 20 sends information indicating the determination result at that moment to the output device 30 (step S405). By doing so, it is possible to prevent the operator of the heavy machinery 40 from receiving incorrectly determined results such as images of scrap metal that were clearly not captured.

[0137] As described above, in the second embodiment, the timing determination unit 532 of the timing determination device 50 may also perform timing determination processing based on control information obtained from the heavy machinery 40, without using the image captured by the imaging device 10. In this case, the timing determination unit 532 may also control the imaging device 10 to perform imaging at the determined timing. The control of the imaging device 10 by the timing determination unit 532 may be performed, for example, by executing a program through the processor of the timing determination device 50.

[0138] Alternatively, the foreign object detection device 20 can also acquire an image of the object to be detected as follows. The image acquisition device 400 stores the images captured by the imaging device 10 in a storage device (e.g., the storage unit 52 of the timing detection device 50, or the storage unit of other information processing devices). The foreign object detection device 20 accesses the storage device and, based on the timing determined by the timing detection device 50, acquires the image of the object to be detected from the images stored in the storage device.

[0139] Furthermore, the "image acquisition timing", "capture timing" and "output timing" described in the second embodiment above will be collectively referred to as "processing timing".

[0140] Furthermore, as a variation from another perspective that can be applied to either the first or second embodiment, in the above description, the image of scrap iron transported by the heavy machinery 40 is used as the judgment object, but the transport of scrap iron can also be carried out by other devices. For example, the judgment system 100 can also be applied in the method of transporting scrap iron using a device such as a bridge crane. In this case, the imaging device 10 can be installed, for example, on a part of a moving crane. Specifically, it includes: a crane body that can travel to both sides in a predetermined direction (travel direction); a trolley that can move in the crane body in a direction orthogonal to the travel direction (lateral direction) on a horizontal plane including the travel direction; and a lifting member provided on the trolley for lifting the transported object, but at least one or more cameras can also be provided on the crane body, the trolley, and the holding member in the lifting member for holding the transported object. That is, the imaging device 10 can be provided on a transport machine, which can also be the heavy machinery 40 or a crane.

[0141] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the specific structure is not limited to these embodiments and may include designs that do not depart from the spirit of the present invention.

[0142] Explanation of reference numerals in the attached figures

[0143] 100: Judgment System

[0144] 10: Filming equipment

[0145] 20: Foreign Object Detection Device

[0146] 30: Output device

[0147] 40: Heavy Machinery

[0148] 401: Movable Body

[0149] 402: Lifting Magnet

[0150] 403: Main Body

[0151] 404: Control Room

[0152] 405: Rotational body

[0153] 406: Driving body

[0154] 21: Input / Output Section

[0155] 22: Storage Department

[0156] 221: Foreign Object Detection Model Storage Unit

[0157] 23: Control Department

[0158] 231: Information Control Department

[0159] 232: Foreign Object Detection Department

[0160] 200: Determination device

[0161] 300: Terminal device

[0162] 51: Input / Output Section

[0163] 52: Storage Department

[0164] 521: Timed determination of model storage unit

[0165] 53: Control Department

[0166] 531: Information Control Department

[0167] 532: Timing Determination Department

[0168] 400: Image acquisition device

Claims

1. A determination system, characterized in that, have: The filming device is mounted on a conveyor used for transporting scrap metal; The foreign object detection device acquires an image of the object to be detected captured by the imaging device and determines whether the scrap iron contains foreign objects.

2. The determination system according to claim 1, characterized in that, The shooting device is mounted on the main body or movable part of the transporter.

3. The determination system according to claim 1 or 2, characterized in that, The camera device captures images of the scrap metal during the transport process by the conveyor.

4. The determination system according to any one of claims 1 to 3, characterized in that, It also includes a timing determination device that determines the processing timing, which can be any timing among the following: image acquisition timing for obtaining an image of the object to be determined, image capture timing for taking a picture of the object to be determined, or output timing for outputting the determination result of the foreign object determination device. When the timing determination device determines that the image acquisition timing is correct, the foreign object detection device acquires the image captured by the imaging device at the specified image acquisition timing. When the timing determination device determines the shooting timing, the shooting device performs shooting at the shooting timing. When the timing determination device determines the output timing, the foreign object determination device outputs the determination result performed at the output timing.

5. The determination system according to claim 4, characterized in that, The timing determination device determines the processing timing based on the image captured by the imaging device or the control information of the conveyor.

6. The determination system according to any one of claims 1 to 5, characterized in that, The foreign object detection device acquires and determines the object by taking an image of the scrap iron being lifted by the conveyor from below.

7. The determination system according to any one of claims 1 to 6, characterized in that, The foreign object detection device acquires an image of the object to be detected while the holding part of the scrap iron lifted by the conveyor is facing the imaging device, and then makes a determination.

8. The determination system according to any one of claims 1 to 7, characterized in that, The foreign object detection device acquires and makes a determination of the object image. The object image is an image taken in a state where the scrap iron, which has been lifted at least once by the conveyor, is placed in an area different from the designated transport destination before being transported to the designated transport destination.

9. A determination method, characterized in that, have: The image capturing step involves capturing images using a camera device installed on a conveyor used for handling scrap metal. The acquisition step involves obtaining an image captured by the imaging device, which serves as the image of the object to be determined. The foreign object determination step uses the image of the object to be determined obtained in the acquisition step to determine whether the scrap iron contains foreign objects.

10. The determination method according to claim 9, characterized in that, In the image capturing step, an image of the scrap metal being lifted by the conveyor is captured from below.

11. The determination method according to claim 9 or 10, characterized in that, In the image capturing step, an image is captured with the holding portion of the scrap metal lifted by the conveyor facing the capturing device.

12. The determination method according to any one of claims 9 to 11, characterized in that, In the image capturing step, images are captured while the scrap metal, which has been lifted at least once by the conveyor, is placed in an area different from the designated transport destination before being transported thereto.

13. A computer program, characterized in that, The computer is used as a foreign object detection device to acquire an image of the object to be detected by a camera device installed on a conveyor for handling scrap iron, and uses the acquired image of the object to be detected to determine whether the scrap iron contains foreign objects.

Images