Waste discrimination system and waste discrimination method

Abstract

Provided are a waste discrimination system and a waste discrimination method capable of improving a discrimination technique for waste. A waste discrimination system includes: a waste site extraction model (221) that extracts a waste site present in a central portion included in a camera image based on the camera image, with reference to a window portion (107) set in advance within the image; a waste discrimination model (222) that screens a grade of waste and a ratio of each grade from a waste image extracted by the waste site extraction model (221), and is generated by including teacher data of a learning image; and an output portion (24) that outputs information of the grade of waste and the ratio of each grade discriminated using the waste discrimination model (222) and based on the waste image.

Description

Technical Field

[0001] This disclosure relates to waste identification systems and methods. Background Technology

[0002] In recent years, from the perspective of effectively utilizing resources, there has been a pursuit of reusing waste materials and other recyclable resources. To reuse waste, it is necessary to identify recyclable resources. A waste identification and processing method that does not rely on human intervention has been proposed, in which images of the waste are pre-generated manually, and a learned model constructed using information about the waste as teacher data and through machine learning is used to determine the waste based on the camera images (e.g., Patent Document 1).

[0003] Patent Document 1: Japanese Patent Application Publication No. 2017-109197

[0004] Patent Document 2: Japanese Patent Application Publication No. 2020-95709

[0005] However, the technology in Patent Document 1 targets dismantled residential buildings and disaster debris for identification, and does not address methods for efficiently identifying waste materials such as metals. For example, iron scrap is traded on the market as a resource for iron recycling, and iron is recovered using electric furnaces. Previously, the grade of waste was determined by visual inspection by workers at the iron scrap processing site. This is because the shredded metal scraps come in various sizes and shapes, requiring visual inspection of the entire material for grading, making automation difficult. Furthermore, the visual inspection method leads to inconsistent results due to varying worker skill levels. Additionally, there are issues related to the aging workforce and personnel availability.

[0006] Patent Document 2 discloses a technique for determining the weight grade of iron scrap. In this method, the scrap is suspended using a magnetic crane, and images taken at the location are used to infer the proportion of the scrap grade based on the suspended portion. This inference process is repeated multiple times, resulting in a final overall determination. However, the technique in Patent Document 2 is limited to situations where the scrap grade is determined while being suspended by a magnetic crane. In actual scrap handling, scrap is often moved in without a crane, making the aforementioned method difficult to apply. Furthermore, in the method of Patent Document 2, the scrap is determined sequentially while being suspended by a magnet, thus requiring a considerable amount of time to determine the grade of all scrap. Therefore, there is room for improvement in techniques for determining the overall shape of the scrap from the obtained images and identifying its grade. Summary of the Invention

[0007] The purpose of this disclosure, which was made in view of this situation, is to provide a waste identification system and a waste identification method that can improve waste identification technology.

[0008] The waste identification system disclosed herein comprises: an acquisition unit that acquires a camera image containing waste; a waste portion extraction model that extracts waste portions existing in the central portion of the camera image based on a window portion pre-set within the image; a waste identification model that filters waste grades and ratios of each grade from the waste image extracted by the waste portion extraction model, and is generated using teacher data including learning images; and an output unit that outputs information on the waste grades and ratios of each grade identified using the waste identification model and based on the waste image.

[0009] Furthermore, the waste identification method disclosed herein uses a waste portion extraction model based on a camera image containing waste to extract waste portions existing in the central part of the camera image, and a waste identification model generated using teacher data containing learning images to identify waste grades and the ratio of each grade. The waste identification method includes the following steps: acquiring a camera image containing the waste; using the waste portion extraction model and extracting a waste image based on the camera image; and outputting information on the waste grades and the ratio of each grade identified using the waste identification model and based on the waste image.

[0010] Furthermore, the waste identification system disclosed herein includes: an acquisition unit that acquires a camera image containing waste; a waste portion extraction model that extracts waste portions existing in the central portion of the camera image based on a window portion pre-set within the image; a foreign object identification model that filters foreign objects other than iron waste from the waste image extracted by the waste portion extraction model, and is generated using teacher data including learning images; and an output unit that outputs information on whether there are foreign objects in the iron waste identified using the foreign object identification model and based on the waste image.

[0011] The waste identification system and method disclosed herein can improve waste identification technology. Attached Figure Description

[0012] Figure 1 This is a diagram showing the outline structure of a waste identification system according to one embodiment of the present disclosure.

[0013] Figure 2 It is a specific example of a camera image of waste that becomes the object of identification.

[0014] Figure 3 It is a specific example of identifying objects outside the main subject that are included in the camera image.

[0015] Figure 4 This is a schematic diagram of the processing of the waste portion of the extracted object.

[0016] Figure 5 This is a diagram showing a specific example of a waste area extracted based on a window.

[0017] Figure 6 This is a specific example containing both the original image and the labeled image of the waste.

[0018] Figure 7 This is a flowchart of the learning phase of the waste extraction model.

[0019] Figure 8 This is a diagram showing the waste parts extracted using the learned waste part extraction model.

[0020] Figure 9 This is a diagram illustrating the learning process of the first waste identification model.

[0021] Figure 10 This is a diagram showing the outline of the identification process performed by the first waste identification model.

[0022] Figure 11 This is a diagram illustrating the learning process of the second waste identification model.

[0023] Figure 12 This is a flowchart illustrating a waste identification method according to one embodiment of the present disclosure.

[0024] Figure 13 It is a standardized conceptual diagram for each group corresponding to the zoom level of the camera image.

[0025] Figure 14 This is an example of a composite image containing a foreign object. Detailed Implementation

[0026] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0027] In each figure, the same or equivalent parts are labeled with the same reference numerals. In the description of this embodiment, descriptions of the same or equivalent parts are appropriately omitted or simplified.

[0028] Figure 1 This is a schematic diagram showing the overall outline of a waste identification system 1 according to an embodiment of the present invention. Hereinafter, the case where the object of identification in this embodiment is iron waste will be described, but the object of identification is not limited to iron waste. For example, the object of identification may also be other metal waste.

[0029] Iron scrap can be broadly categorized into two types based on its source. One type is processing scrap (also known as factory-generated scrap) generated during the manufacturing process. After being collected by recyclers, this processing scrap is renamed as leftover end stock, steel shavings, or iron filings and circulated. Furthermore, most of this processing scrap is not processed (intermediate processing) and is taken back by steel manufacturers. In other words, processing scrap is iron scrap with a traceable origin, and like recycled scrap, it is scrap with good usability in terms of quality. Additionally, the possibility of foreign matter contamination during the generation, recycling, and transportation stages is low.

[0030] Another type of waste is aging waste generated from the aging of steel structures. Aging waste also includes waste generated during the repair or damage phase. It is generated in various locations, such as during building dismantling, equipment replacement, and after use, including vehicles and containers, and its shape varies widely. Therefore, after recycling, aging waste undergoes shaping, crushing, and volume reduction processing to improve steelmaking efficiency before being treated as heavy scrap. Furthermore, for steel plates from household appliances, automobile bodies, vending machines, etc., the primary process is to reduce volume through crushing and then use magnetic separation to remove only the iron. These aging wastes vary in their generation, recycling, and processing stages; therefore, they are graded after processing. The grading of aging waste is determined by its shape, including thickness, width, and length. Currently, the unified specifications for iron scrap inspection established by the Japan Iron Scrap Association (Company) in 1996 are widely used.

[0031] As mentioned above, conventionally, the grade of scrap metal is identified at the scrap metal processing site by visual observation of the workers. Furthermore, this method suffers from inconsistencies in identification results due to varying levels of worker skill. The scrap metal identification system 1 described in this embodiment addresses these problems. In summary, it identifies scrap metal based on camera images captured from the scrap metal, replacing visual observation by the workers.

[0032] In this embodiment, an example is described where the scrap is also classified into six categories: HS, H1, H2, H3, and L1 and L2, which have low iron content, such as rusted galvanized sheet. However, the grade of the scrap being classified is not limited to these. Furthermore, the grade of the scrap being classified may also include residual end material scrap (shearing scrap), iron filings (chips), etc. In other words, the grade of scrap classified in this embodiment can include any grade of scrap to match the needs of the manufacturing site.

[0033] like Figure 1As shown, the waste identification system 1 according to this embodiment includes multiple cameras 10 and an information processing device 20. The multiple cameras 10 and the information processing device 20 are connected by a network 30. The multiple cameras 10 are, for example, network cameras, and the camera images captured are sent to the information processing device 20 via the network 30. Figure 1 The example shown has four cameras 10, but the number of cameras 10 is not limited to this. There may be fewer than four cameras 10, or even just one. Alternatively, there may be more than four cameras 10. The camera images are taken of the scrap metal at a point in time after it has been transported by truck and temporarily moved to the marshalling yard. Figure 2 This refers to a specific example of a camera image captured by camera 10. For example... Figure 2 As shown, the camera image contains a mixture of iron scrap of various grades. In summary, the information processing device 20 identifies the grades of the iron scrap in the camera image and the ratio of each grade based on multiple models generated through machine learning. In actual operation, scrap is traded based on the weight ratio of each grade and the total weight. Therefore, in this embodiment, the information processing device 20 will be described in terms of identifying the ratios related to the weight of each grade of scrap in the camera image.

[0034] The information processing device 20 includes a control unit 21, a storage unit 22, an acquisition unit 23, and an output unit 24.

[0035] The control unit 21 includes at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (central processing unit) or a dedicated processor specifically designed for certain processing. The dedicated circuit is, for example, an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). The control unit 21 controls the various parts of the information processing device 20 while performing processing related to the operation of the information processing device 20.

[0036] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, RAM (random access memory) or ROM (read-only memory). RAM is, for example, SRAM (static random access memory) or DRAM (dynamic random access memory). ROM is, for example, EEPROM (electrically erasable programmable read-only memory). The storage unit 22 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores data for the operation of the information processing device 20, and data obtained through the operation of the information processing device 20. For example, the storage unit 22 stores a waste extraction model 221 and a waste identification model 222. The waste identification model 222 includes a first waste identification model 222-1, a second waste identification model 222-2, and a selection model 222-3.

[0037] The waste identification system 1 uses camera images to classify waste into grades, replacing the visual inspection by operators. However, judgment errors sometimes occur due to camera images. When classifying waste by camera images after handling, it is practically difficult to only photograph the waste to be identified, resulting in objects other than the identified object being included in the camera image. These objects other than the identified object include, for example, the ground, the background, and waste other than the identified object. Figure 3 This is an example of a camera image that includes objects other than the target being identified. For example... Figure 3 As shown, the camera image includes the ground 101, the background 102, waste 103 and 104 other than the waste to be identified, and the transport vehicle 105 (truck 105). The objects in the camera image other than the identified object are not limited to these; they also include nearby waste disposal sites entering the frame. This means that even if the camera intends to only capture waste, the surrounding conditions can make it difficult, leading to errors in the identification process.

[0038] To reduce such errors, the inventors repeatedly studied the construction of a system for determining the grade of waste based on machine learning algorithms such as neural networks (described later). As a result, the inventors discovered that waste grade can be determined with high accuracy through the following two processes: The first process is the extraction of waste portions from camera images. The second process is the identification of the grade of the waste portions extracted in the first process. In summary, the waste identification system 1 performs the first process using a waste portion extraction model 221 and the second process using a waste identification model 222.

[0039] First, the first process will be described. After a camera image of the waste material for which a grade determination is to be made is sent to the information processing device 20 via network 30, the waste part extraction model 221 extracts the waste parts from such camera images. In this embodiment, the method by which the waste part extraction model 221 extracts waste parts uses semantic segmentation as one of the spatial classification processes.

[0040] Semantic segmentation is a method of classifying pixels into categories based on their meaning (information from surrounding pixels) (Japanese Patent Application Publication No. 2020-21188, Badrinalayanan, V., A. Kendall, and R. Cipolla. Segnet: A deep convolutional encoder decoder architecture for imagesegmentation. arXiv. Preprint arXiv:1511.0051). Semantic segmentation is used, for example, in autonomous driving systems to investigate appearances (roads, buildings), shapes (vehicles, pedestrians), and to understand the spatial relationships (order) between different categories such as roads and sidewalks.

[0041] use Figure 4 This section outlines the process of extracting waste portions of objects from camera images using semantic segmentation to determine their grade. The camera images include waste images from the central portion of the image, which are then graded. Here, as... Figure 4 As shown, when extracting the waste portion, a window 107 pre-set within the camera image 106 is used to determine the waste of the target object. This window 107 is a region located in the waste image, extending from the image center along both the short and long sides of the camera image at approximately 1 / 2 (1 / 4) the distance from the image center. In other words, the window 107 is a rectangular region with lengths of 1 / 4M and 1 / 4L respectively in the long and short directions from the image center, assuming the lengths of the long and short sides of the camera image are set to M and L respectively. Furthermore, as... Figure 4As shown, in this embodiment, a waste portion 108 originating from the waste within the window 107 is extracted. "A waste portion originating from the waste within the window" refers to a waste portion within the window where at least a portion of a collection of waste materials (hereinafter also referred to as a waste group) exists. In other words, if a portion of a waste group exists within the window 107, the waste materials contained in that waste group are treated as a waste portion originating from the waste within the window.

[0042] Figure 5 This illustrates a specific example of a window 107 that serves as a reference when extracting waste portions from an image, and a waste portion used for determining the extraction level. As described above, waste portions are constructed into waste groups of arbitrary shapes, starting from the waste present in the window. Figure 5 In Examples 1 to 5, the parts 109 to 113 represent the parts of the extracted waste material.

[0043] In this semantic segmentation-based extraction process, as illustrated in the autonomous driving example above, multiple objects (roads, buildings, vehicles, pedestrians) contained in the image can be classified separately. Therefore, in this embodiment, it is also possible to extract (classify) all waste materials contained in the original image from objects other than waste materials. However, in this embodiment, in order to extract only newly moved-in waste materials that should be graded, waste materials that do not originate from the waste materials within the window are treated as background. This differs from typical semantic segmentation.

[0044] To perform the aforementioned extraction process using the waste extraction model 221, the parameters of the waste extraction model 221 are learned using multiple camera images. First, camera images containing waste are prepared, assuming they will be captured during waste grading. Preferably, the camera images contain various types of waste assuming a location where waste grading is being conducted, and multiple images with various backgrounds exist. For the image data containing each type of waste, the operator visually labels the waste group used for grading and other parts (background, etc.) with different colors, creating labeled images. Figure 6 This represents an example of an original image containing waste and a labeled image. These original images and the labeled image correspond to teacher data. In other words, the waste extraction model 221 is a learning model generated using a combination of the original image containing waste and the labeled image as teacher data. Figure 6 As shown, the labeled image includes labels (labels 114, 115) representing regions indicating waste areas and labels (labels 116, 117) representing other areas. Using this teacher data, parameter learning (learning phase) is performed for the waste area extraction model 221.

[0045] Figure 7This flowchart illustrates the learning phase of the waste extraction model 221. First, a combination of camera images (original waste images) and corresponding label images is prepared as teacher data (steps S1, S2). Furthermore, a sufficient number of label images (e.g., approximately 1000) is preferably used to enable high-precision waste extraction. The original waste images and label images are divided into a training set for parameter learning and a test set for network accuracy evaluation (step S3). Using the training set for parameter learning, the parameters of the waste extraction model 221 are learned (step S4). Then, using the test set for accuracy evaluation, the network is trained and evaluated (step S5), and the parameters are adjusted again. Based on this result, the network parameters are determined, and the network is trained (step S6). Furthermore, if an error occurs due to a background completely different from the background presented in the 1000 images, a new image containing that completely different background can be used for retraining. As a result of the above training, the waste portion present in the central part of the image can be extracted from the image containing waste. Figure 8 This illustrates an example of extracting waste areas from camera images using the learned waste area extraction model 221. For example... Figure 8 As shown, waste portion 118 is extracted from the other area 119.

[0046] After the waste extraction process using semantic segmentation is executed by waste extraction model 221, a second process is executed, namely, the process of identifying the level of the waste extracted by the first process. In this embodiment, the second process is executed by the first waste identification model 222-1, the second waste identification model 222-2, and the selection model 222-3.

[0047] The first scrap identification model 222-1 is a learning model that identifies the grade and ratio of scrap contained in a scrap image based on the scrap image itself. The first scrap identification model 222-1 is generated from teacher data containing a first learning image. The first learning image is an image of iron scrap of a single grade. That is, the first scrap identification model 222-1 is generated based on teacher data containing the first learning image and performance data related to the identification of that first learning image, and through machine learning algorithms such as neural networks. Figure 9 This represents a summary of the learning process of the first waste identification model 222-1. For example... Figure 9As shown, the first learning image, namely an image of iron scrap of a single grade, is input into the input layer of the first scrap identification model 222-1. For each image of iron scrap of a single grade, a correspondence is established between the performance data of the grade identified by the operator as performance data. The weighting factors between neurons are adjusted using this teacher data, thereby performing the learning process of the first scrap identification model 222-1. Furthermore, Figure 9 Examples of correspondences between images and HS, H1, H2, H3, L1, and L2 are shown, but the grades are not limited to these. Images of iron scrap from a single grade can also be images of iron scrap from any other grade. Here, Figure 9 The diagram shows an example of a first waste identification model 222-1 generated based on a multilayer perceptron consisting of an input layer, hidden layers, and an output layer, but it is not limited to this. The first waste identification model 222-1 can be a model generated by any other machine learning algorithm. For example, the first waste identification model 222-1 can also be a model generated based on machine learning algorithms such as Convolutional Neural Network (CNN) and deep learning.

[0048] When identifying the grade and ratio of waste contained in a waste image using the first waste identification model 222-1, the control unit 21 uses the first waste identification model 222-1 and identifies the ratio of waste based on the area ratio of each grade of waste in the waste image. Figure 10 This represents a summary of the identification process performed by the first waste identification model 222-1. For example... Figure 10 As shown, the control unit 21 inputs images of each portion of a grid-like waste image to the first waste identification model 222-1 to identify the grade of waste in that local image. Thus, by randomly extracting local images of the grid-like waste image and identifying the waste grade, the control unit 21 calculates the area ratio of the waste grades in that image. Furthermore, the control unit 21 converts the area ratio into a weight ratio based on the volume density of each waste material. In this way, the control unit 21 calculates the grades of waste contained in the waste image and the ratio of each grade. Furthermore, this example shows the control unit 21 randomly extracting a portion of a local image of the waste image to calculate the aforementioned area ratio, but it is not limited to this. For example, the control unit 21 could also calculate the aforementioned area ratio based on all local images of the waste image.

[0049] The second scrap identification model 222-2 is a learning model that identifies the grade and ratio of scrap contained in a scrap image based on the scrap image itself. The second scrap identification model 222-2 is generated using teacher data containing a second learning image that differs from the first learning image. The second learning image is an image of mixed-grade iron scrap. Mixed-grade iron scrap is iron scrap containing multiple grades. In other words, the second scrap identification model 222-2 is generated using machine learning algorithms such as neural networks, based on teacher data containing the second learning image and the performance data related to the identification of that second learning image. Figure 11 This shows an overview of the learning process of the second waste identification model 222-2. (See diagram below.) Figure 11 As shown, the second learning image, namely an image of mixed-grade iron scrap, is input into the input layer of the second scrap identification model 222-2. For each mixed-grade iron scrap image, a correspondence is established between the grade identified by the operator as actual data and the ratio of each grade. The weighting factors between neurons are adjusted using this teacher data for model learning. Here, Figure 11 The second waste identification model 222-2 illustrated in the diagram is an example of a model generated based on a multilayer perceptron consisting of an input layer, a hidden layer, and an output layer, but it is not limited to this. The second waste identification model 222-2 can be a model generated using any other machine learning algorithm. For example, the second waste identification model 222-2 can also be a model generated based on machine learning algorithms such as Convolutional Neural Network (CNN) or deep learning. When identifying the grade and ratio of waste contained in a waste image using the second waste identification model 222-2, similar to the first waste identification model 222-1, local images segmented from the waste image are randomly extracted to identify the waste grade, and the control unit 21 calculates the weight ratio of the waste grades in that image. This is because the accuracy of the determination can be improved by segmenting the image and repeatedly identifying randomly selected images. Furthermore, here is an example of calculating the weight ratio of the waste grade by randomly extracting a portion of the camera image, but it is not limited to this. In this respect, it is the same as the first waste identification model. The control unit 21 can also calculate the above weight ratio based on all the local images of the waste image and by segmenting the image.

[0050] Model 222-3 is a model that infers whether either the first waste identification model 222-1 or the second waste identification model 222-2 outputs a more reliable solution when identifying the grade and ratio of waste contained in a waste image based on the waste image. Model 222-3 selects the model that outputs the more reliable solution based on the inference result. Furthermore, the control unit 21 uses the model selected by model 222-3 to identify the grade and ratio of waste based on the waste image. In other words, model 222-3 determines whether either the first waste identification model 222-1 or the second waste identification model 222-2 is used for the identification of waste grades based on camera images. The teacher data involved in selection model 222-3 includes a waste image containing waste acquired from camera 10 via network 30, the waste grades and ratios inferred by the first waste identification model 222-1, the waste grades and ratios inferred by the second waste identification model 222-2, and the grades and ratios identified by the operator as performance data. The performance data involved in model selection is determined based on the identification results when input to the first waste identification model 222-1 and the second waste identification model 222-2, respectively, and the results of the grades and ratios identified by the operator in relation to the waste image. In other words, selection model 222-3 is an inference model generated using such teacher data through machine learning algorithms such as neural networks. Selection model 222-3 is generated, for example, based on machine learning algorithms such as multilayer perceptrons, convolutional neural networks (CNNs), and deep learning.

[0051] The acquisition unit 23 acquires camera images containing waste from the camera 10 via the network 30. The acquisition unit 23 includes at least one communication interface. The communication interface may be, for example, a LAN interface, a WAN interface, an interface corresponding to mobile communication standards such as LTE (Long Term Evolution), 4G (4th generation), or 5G (5th generation), or an interface corresponding to short-range wireless communication such as Bluetooth (registered trademark). The acquisition unit 23 receives data for the operation of the information processing device 20 and transmits the data obtained through the operation of the information processing device 20.

[0052] The output unit 24 includes at least one output interface. The output interface is, for example, a display. The display is, for example, an LCD (liquid crystal display) or an organic EL (electroluminescence) display. The output unit 24 outputs data obtained through the operation of the information processing device 20. Alternatively, the output unit 24 may replace the information processing device 20 in this form and be connected to the information processing device 20 as an external output device. As a connection method, any method such as USB, HDMI, or Bluetooth can be used.

[0053] The functions of the information processing device 20 are implemented by the processor, which corresponds to the control unit 21, executing the program involved in this embodiment. That is, the functions of the information processing device 20 are implemented by software. The program causes the computer to perform the actions of the information processing device 20, thereby enabling the computer to function as the information processing device 20. In other words, the computer functions as the information processing device 20 by executing the actions of the information processing device 20 according to the program.

[0054] In this embodiment, the program can be pre-recorded on a computer-readable recording medium. Computer-readable recording media include non-transitory computer-readable media, such as magnetic recording devices, optical discs, optical-magnetic recording media, or semiconductor memory. Program distribution is achieved, for example, through the sale, transfer, or lending of portable recording media such as DVDs (digital versatile discs) or CD-ROMs (compact disc read-only memory) containing the program. Alternatively, program distribution can also be achieved by pre-storing the program in the server's storage and sending the program from the server to other computers. Furthermore, the program can also be provided as a program product.

[0055] In this embodiment, the computer temporarily stores, for example, a program recorded on a portable recording medium or a program sent from a server, in the main storage device. The computer then uses its processor to read the program stored in the main storage device and executes processing based on the read program. Alternatively, the computer may directly read the program from the portable recording medium and execute processing based on the program. Or, the computer may sequentially execute processing based on the received program each time it receives a program from the server. Alternatively, processing may be performed using a so-called ASP (Application Service Provider) type service, where the program is not sent from the server to the computer, but the function is achieved solely through executing instructions and obtaining results. The program contains information for processing by the computer and is information that conforms to the program. For example, data that is not a direct instruction to the computer but has the nature of specifying the computer's processing can also be considered "information that conforms to the program."

[0056] Some or all of the functions of the information processing device 20 can also be implemented by a dedicated circuit equivalent to the control unit 21. That is, some or all of the functions of the information processing device 20 can also be implemented by hardware.

[0057] Next, a waste identification method performed by the waste identification system 1 according to an embodiment of the present disclosure will be described. Figure 12 This is a flowchart illustrating a waste identification method according to one embodiment of the present disclosure.

[0058] First, the camera 10 of the waste identification system 1 captures a camera image containing waste (step S10). Next, the camera 10 sends the camera image to the information processing device 20 via the network 30. The acquisition unit 23 of the information processing device 20 acquires the camera image via the network 30 (step S20).

[0059] Next, the control unit 21 extracts the area of ​​the photographed waste from the acquired camera image (step S30). The process of extracting the waste becomes... Figure 4 As shown, semantic segmentation and other methods are used to extract the image of the moved-in waste material that exists in the center of the image.

[0060] Next, based on the image of the determined waste area, i.e. the waste image, the control unit 21 uses the selection model 222-3 to identify whether to use the first waste identification model 222-1 or the second waste identification model 222-2 (step S40).

[0061] Next, the control unit 21 uses the model selected by the selection model 222-3 from the first waste identification model 222-1 or the second waste identification model 222-2 to identify the grade and ratio of waste contained in the camera image (step S50).

[0062] Next, the control unit 21 outputs the waste grade and ratio identified in step S40 to the output unit 24. The output unit 24 outputs the waste grade and ratio identified in step S40 (step S60).

[0063] Thus, according to an embodiment of the waste identification system 1 of this disclosure, from a camera image of the waste captured by camera 10, a specific part of the imported waste located at the center of the camera image is determined and the waste image is extracted. Using this waste image, the grade and ratio of the waste can be automatically identified using either the first waste identification model 222-1 or the second waste identification model 222-2. Furthermore, the selection model 222-3 automatically selects the more appropriate model, choosing between the first waste identification model 222-1 and the second waste identification model 222-2. In other words, according to an embodiment of the waste identification system 1 of this disclosure, the grade and ratio of the waste can be identified and output without manual intervention. In other words, according to an embodiment of the waste identification system 1 of this disclosure, waste identification technology can be improved.

[0064] Furthermore, the waste part extraction model 221 of the waste identification system 1 according to one embodiment of this disclosure uses semantic segmentation to extract waste parts. When comparing the semantic segmentation-based method with methods using other object detection methods, the semantic segmentation method has advantages in the following aspects.

[0065] (1) Semantic segmentation labels pixels, thus achieving higher accuracy than other object detection methods.

[0066] (2) Object detection requires objects to be contained within bounding boxes, while semantic segmentation can clearly capture objects with irregular shapes. A method is disclosed in Japanese Patent Application Publication No. 2020-95709 that extracts crane parts from images containing waste using object detection models such as YOLO, but this method sets an extraction box in the image for detection.

[0067] In this embodiment, the portion of the waste image that is used for grade determination has an irregular shape and needs to be separated from objects of various shapes that serve as the background. Therefore, object detection methods are not applicable. By using semantic segmentation, the waste image can be extracted with higher accuracy. As described above, an image containing waste is captured, and the waste present in the center of the image is extracted using semantic segmentation. The extracted image is then used to determine the waste grade, thereby enabling a rapid one-time determination of the waste that has been moved in.

[0068] The system described above can be used to determine the grade of scrap, but the same system can also be applied to detect foreign objects (including substances other than iron) in the scrap. That is, foreign objects are objects other than iron contained in iron scrap, such as motors, wood chips, tires, and various other foreign objects. It is desirable to have as few foreign objects as possible when melting scrap. Especially when directly melting foreign objects, if they are dissolved in iron, residual elements that cannot be removed remain in the molten steel. Examples of residual elements include Cu, Sn, Cr, and Ni. It is well known that high Cu levels can cause damage during hot rolling.

[0069] Currently, operators visually inspect waste materials while classifying them, removing any foreign objects found. However, visual inspection alone is insufficient to completely detect foreign objects. Furthermore, in order to enable the system to automatically classify waste materials, it is desirable to achieve unmanned, automated detection of foreign objects.

[0070] As a machine learning-based detection system, it can detect foreign objects using the same approach as the scrap classification system. Specifically, after capturing images with a camera, scrap portions are detected from the images through semantic segmentation. Then, foreign objects are detected. The system makes decisions using the same logic as the second scrap identification model 222-2. That is, by pre-learning images containing both scrap and foreign objects, it can detect foreign objects within the scrap. The foreign object identification model is generated using machine learning algorithms such as neural networks, based on teacher data containing training images related to scrap images containing foreign objects (images of foreign objects mixed into scrap) and actual data on foreign objects involved in those training images.

[0071] However, identifying foreign objects presents significant challenges. Examples of problematic foreign objects mixed into scrap are rare. Therefore, obtaining a large number of images containing both scrap iron and foreign objects for learning purposes is extremely difficult. Furthermore, even with a large accumulation of images for learning, the types of foreign objects change over time. Therefore, a method is used to artificially create images containing both scrap iron and foreign objects through data augmentation. Data augmentation refers to expanding the learning data by applying certain processing techniques to the original learning data.

[0072] The following is an example of a method for synthesizing an image of the object to be detected and a background image. Here, the following process was performed for the case of creating an image of iron scrap containing a large amount of copper, which is the subject of the problem. Furthermore, an electric motor (hereinafter, simply referred to as a "motor") is used as an example of a foreign object containing copper.

[0073] (1) Prepare images of the foreign object unit (motor). In other words, prepare images of multiple general motors.

[0074] (2) Imagine the state of the motor mixed with waste materials, delete the local shape of the motor, or change the local color, etc., to increase the changes in the image of the motor.

[0075] (3) Create an artificial image that combines the waste image and the motor image mentioned above.

[0076] Here, regarding the process (3) described above, even if the image of the detected object, such as the motor, is composited onto the background image of the scrap iron, the image of the motor will appear to float from the background depending on the circumstances, thus requiring the generation of an image assimilated into the background image. Therefore, by using Poisson image editing for compositing, it is possible to suppress the misidentification of composite objects other than the motor. Poisson image editing is an image editing method proposed by M. Prez et al. (Patrick Perez, Michel Gangnet, and Andrew Blake, "Poisson image editing", Association for Computing Machinery vol.22,3,313-318, 2003). Poisson image editing is an image processing method that can produce composite images without inconsistencies by calculating the simultaneous equations of the Poisson equations related to the gradient of the image.

[0077] Figure 14 An example of a composite image is shown, containing scrap iron from a motor manufactured after the above processing. Figure 14 In the image, a motor is present as a foreign object within the area enclosed by a circle. Since Poisson image editing can generate any number of images, 30,000 teacher data points were extracted from the waste and motor images for training. After validating the waste identification model using this teacher data with 5,000 images, it was confirmed that the motor (foreign object) could be detected in 90% of cases.

[0078] This disclosure has been described based on the accompanying drawings and embodiments, but it should be noted that various modifications and alterations can be easily made based on this disclosure by those skilled in the art. Therefore, it should be understood that such modifications and alterations are included within the scope of this disclosure. For example, the functions contained in each structure or step can be reconfigured logically without contradiction, and multiple components or steps can be combined into one or divided.

[0079] For example, in this embodiment, the method of extracting waste parts by semantic segmentation for the waste part extraction model 221 is described, but the method of extracting waste parts is not limited to this, and any spatial classification processing can be used.

[0080] Furthermore, in this embodiment, for example, an example is shown as the waste identification model 222 comprising a first waste identification model 222-1, a second waste identification model 222-2, and a selection model 222-3. Thus, by using a model comprising the first waste identification model 222-1, the second waste identification model 222-2, and the selection model 222-3 as the waste identification model 222, high-precision waste grade determination can be performed. However, the waste identification model 222 is not limited to this. For example, the waste identification model 222 can also be constructed using either the first waste identification model 222-1 or the second waste identification model 222-2. In other words, the waste identification system 1 can also use only the first waste identification model 222-1 generated by machine learning using machine learning algorithms such as neural networks, based on teacher data containing images of a first learning image (i.e., a single grade of iron scrap) and performance data related to the identification of that first learning image. Alternatively, the scrap identification system 1 may use only teacher data based on the performance data of the identification involving images of mixed-grade iron scrap and the second learning image, and generate a second scrap identification model 222-2 using machine learning algorithms such as neural networks. In other words, in this embodiment, a system that uses the first scrap identification model 222 and the second scrap identification model 222-2 to determine the scrap grade has been described, but it is also possible to use either the first scrap identification model 222-1 or the second scrap identification model 222-2 to identify the scrap grade. In this case, step S40 described above is omitted. Moreover, when using either the first scrap identification model 222-1 or the second scrap identification model 222-2, it is desirable to have sufficient teacher data in advance containing each learning image and the performance data of the identification involved in the learning image to improve the accuracy of grade determination. Alternatively, images can continue to be captured after the system is running, while simultaneously acquiring the operator's judgment results and relearning, thereby improving accuracy.

[0081] Furthermore, for example, in the learning and discrimination processes of the first waste identification model 222-1, the second waste identification model 222-2, and the selection model 222-3, the control unit 21 may also use scaling information corresponding to each image. When using scaling information, the camera 10 sends the scaling information of the ONVIF data corresponding to the camera image, along with the camera image, to the information processing device 20 via the network 30. For example, the first learning image, the second learning image, and the waste image may also be normalized based on the scaling information corresponding to each image. In other words, the control unit 21 normalizes each image at a predetermined magnification based on the scaling information corresponding to the first learning image, the second learning image, and the waste image, respectively. Moreover, the control unit 21 performs learning processing using the normalized first learning image and the second learning image, and also performs discrimination processing based on the waste image. By normalizing each image through such normalization processing, the discrimination accuracy of the waste-based identification system 1 can be improved.

[0082] Here, when each image is normalized to a predetermined magnification based on the scaling information, the control unit 21 can also classify the images into groups based on the scaling information and normalize each group with a different magnification. Figure 13 A standardized conceptual diagram representing each set of waste images. Figure 13 In this study, waste images are classified based on magnification. Specifically, waste images with a magnification range of x0 or higher but less than x1 are classified. 01 (Hereafter, also referred to as the first range R) 01 In the case of (x1 and x2), the waste image is classified as Group 1. When the magnification of the waste image is above x1 but below x2, R... 12 (Hereafter, also referred to as the second range R) 12 In the case of a waste image being classified as Group 2, where the magnification of the waste image is greater than x2 but less than x3, R... 23 (Hereafter, also referred to as the 3rd range R) 23 In the case of [missing information], the waste image is classified as Group 3. The magnification of the waste image is in the range R [missing information], which is greater than x3 but less than x4. 34 (Hereafter, also referred to as the 4th range R) 34 In the case of [missing information], the waste image is classified as Group 4. The magnification of the waste image is in the range R [missing information], which is greater than x4 but less than x5. 45 (Hereafter, also referred to as the 5th range R) 45 In this case, the waste image was classified as Group 5. Furthermore, the first range R... 01 , second range R 12 , third range R 23 , 4th range R 34 and the 5th range R 45Each waste image in the image is normalized to a reference magnification X for its respective range. 01 X 12 X 23 X 34 and X 45 In other words, the first learning image, the second learning image, and the waste image are standardized to different magnifications based on the scaling information corresponding to each image. In other words, the control unit 21 performs standardization using any one of multiple reference magnifications determined based on the scaling information of the waste image. This suppresses inconsistencies in image resolution caused by over-magnification or under-magnification of the waste image, thereby improving the recognition accuracy of the waste-based identification system 1. Furthermore, Figure 13 The image shown is an example where the control unit 21 classifies the image into 5 groups and standardizes it with 5 corresponding magnifications, but it is not limited to this. For example, the control unit 21 may also classify the number of groups into 4 or less or 6 or more, and standardize them with different magnifications respectively.

[0083] Furthermore, the above description illustrates an example of using image scaling information for learning and discrimination processing, but is not limited to this. For example, the waste identification system 1 can also use at least a portion of the ONVIF data obtained from the camera 10 for learning and discrimination processing. The ONVIF data contains translation, tilt, and scaling information. That is, the waste identification system 1 can also use at least any one of the translation, tilt, and scaling information for learning and discrimination processing.

[0084] Furthermore, for example, in the learning and identification processes of the first waste identification model 222-1, the second waste identification model 222-2, and the selection model 222-3, the control unit 21 may also additionally use information related to the waste receiving companies. In this way, the identification accuracy of the waste identification system 1 can be improved by considering the trends of waste received by each receiving company.

[0085] Furthermore, for example, the waste identification system 1 can also accumulate the images used for identification after the identification process as new teacher data. Moreover, the control unit 21 can relearn the first waste identification model 222-1, the second waste identification model 222-2, and the selection model 222-3 based on these images, combined with the results of the operator's identification of the grade and the ratio of each grade. For example, assuming there is a problem with the output result (identification result), the information of the problematic output, along with the corresponding image and performance data, can be used as teacher data to relearn at least one of the first waste identification model 222-1, the second waste identification model 222-2, and the selection model 222-3. This improves the accuracy and speed of identification performed by the first waste identification model 222-1, the second waste identification model 222-2, and the selection model 222-3.

[0086] Furthermore, for example, in this embodiment, the camera image is an image taken at a point in time after the scrap metal has been transported by truck and temporarily moved to the shunting yard, but it is not limited to this. For example, the camera image could also be an image taken at the manufacturing site where the scrap metal is being lifted by a crane. In this case, a lighting device that illuminates the scrap metal during the shooting can also be used. This allows for the selection of a clear camera image. In either case, after the scrap extraction operation is further determined from the captured image using the methods already described, the scrap grade determination system can be operated, thereby improving the accuracy of the scrap grade determination.

[0087] Furthermore, in this embodiment, for example, the window 107 is a region covering 1 / 4 of the overall image with the center of the camera image as a reference (when the lengths of the long and short sides of the camera image are set to M and L respectively, it is a rectangular area extending 1 / 4M and 1 / 4L from the center of the image along the long and short sides respectively), but it is not limited to this. The window 107 can be any range as long as it is a part of the overall image with the center of the camera image as a reference. For example, the window 107 could also be a rectangular area extending 1 / 3M and 1 / 3L from the center of the image with the lengths of the long and short sides of the camera image set to M and L respectively.

[0088] Explanation of reference numerals in the attached figures

[0089] 1...Waste identification system; 10...Camera; 101...Ground; 102...Background; 103, 104...Waste; 105...Transport vehicle (truck); 106...Camera image; 107...Window; 108-113, 118...Waste location; 114-117...Label; 119...Area; 20...Information processing device; 21...Control unit; 22...Storage unit; 23...Acquisition unit; 24...Output unit; 221...Waste location extraction model; 222...Waste identification model; 222-1...First waste identification model; 222-2...Second waste identification model; 222-3...Selection model; 30...Network.

Claims

1. A waste identification system, characterized in that, have: The acquisition unit acquires camera images containing waste. The waste part extraction model is based on the camera image and uses a pre-set window in the image as a reference to extract the waste part existing in the central part of the camera image. The waste identification model filters the waste grade and the ratio of each grade from the waste images extracted by the waste part extraction model, and is generated using teacher data that includes images for learning purposes; as well as The output unit outputs information about the grade of waste and the ratio of each grade, identified using the waste identification model and based on the waste image. The window defines a 1 / 4 portion of the overall image, with the center of the camera image as the reference. Semantic segmentation is used to extract waste areas starting from the waste within this window. The semantic segmentation method is a method that uses computer processing of the camera image to classify pixels based on information from surrounding pixels. It includes steps to extract waste areas originating from the waste within the window and to treat waste areas not originating from the waste within the window as background. The "waste area starting from the waste within the window" refers to at least a portion of the waste aggregate formed by multiple waste materials existing within the waste area of ​​the window.

2. The waste identification system according to claim 1, characterized in that, The learning image is an image of iron scrap of a single grade. When using the scrap identification model to identify the grade of scrap contained in the scrap image and the ratio of each grade, the ratio is identified based on the area ratio of each grade of scrap in the scrap image.

3. The waste identification system according to claim 1, characterized in that, The images used for learning are images of mixed grades of iron scrap.

4. The waste identification system according to claim 1, characterized in that, The waste identification model includes: The first waste identification model is based on the waste image to identify the grade and ratio of waste contained in the camera image, and is generated using teacher data containing the first learning image. The second waste identification model, based on the waste image, identifies the grade and ratio of waste contained in the camera image, and is generated using teacher data containing a second learning image different from the first learning image; and The selection model, based on the waste image, determines whether to use either the first waste identification model or the second waste identification model. The output unit will output information about the waste grade and the ratio of each grade, which is determined by the selection model in the first waste identification model or the second waste identification model, based on the waste image.

5. The waste identification system according to claim 4, characterized in that, The first learning image is an image of iron scrap of a single grade. When using the first scrap identification model to identify the grade of scrap contained in the scrap image and the ratio of each grade, the ratio is identified based on the area ratio of each grade of scrap in the scrap image.

6. The waste identification system according to claim 4 or 5, characterized in that, The second learning image is an image of mixed-grade iron scrap.

7. The waste identification system according to claim 4 or 5, characterized in that, The first learning image, the second learning image, and the waste image are standardized based on scaling information corresponding to each image.

8. The waste identification system according to claim 4 or 5, characterized in that, At least one of the first waste identification model, the second waste identification model, and the selection model is relearned based on the waste image and the information output by the output unit.

9. A waste identification method, comprising a waste extraction model based on a camera image containing waste, using a pre-defined window within the image as a reference to extract waste portions existing in the central portion of the camera image, and a waste identification model that identifies waste grades and the ratio of each grade, and is generated using teacher data containing training images. The waste identification method is characterized by including the following steps: Acquire camera images containing the waste; The waste portion is used to extract a model, and the waste image is extracted based on the camera image; and The system will output information on the waste grade and the ratio of each grade, identified using a waste identification model based on the waste image. The window defines a 1 / 4 portion of the overall image, with the center of the camera image as the reference. Semantic segmentation is used to extract waste areas starting from the waste within this window. The semantic segmentation method is a method that uses computer processing of the camera image to classify pixels based on information from surrounding pixels. It includes steps to extract waste areas originating from the waste within the window and to treat waste areas not originating from the waste within the window as background. The "waste area starting from the waste within the window" refers to at least a portion of the waste aggregate formed by multiple waste materials existing within the waste area of ​​the window.

10. A waste identification system, characterized in that, have: The acquisition unit acquires camera images containing waste. The waste part extraction model is based on the camera image and uses a pre-set window in the image as a reference to extract the waste part existing in the central part of the camera image. The foreign object identification model filters out foreign objects other than iron scrap from the scrap images extracted by the scrap extraction model, and is generated using teacher data that includes images for learning purposes. as well as The output unit outputs information regarding the presence or absence of foreign objects in the scrap metal, as determined using the foreign object detection model and based on the scrap image. The window defines a 1 / 4 portion of the overall image, with the center of the camera image as the reference. Semantic segmentation is used to extract waste areas starting from the waste within this window. The semantic segmentation method is a method that uses computer processing of the camera image to classify pixels based on information from surrounding pixels. It includes steps to extract waste areas originating from the waste within the window and to treat waste areas not originating from the waste within the window as background. The "waste area starting from the waste within the window" refers to at least a portion of the waste aggregate formed by multiple waste materials existing within the waste area of ​​the window.

11. The waste identification system according to claim 10, characterized in that, The foreign object image and the scrap image are synthesized using an image editing method that uses Poisson image editing. The foreign object image is then mixed into the scrap image and used as the learning image for the foreign object identification model.

Images