Method and apparatus for providing scrap iron classification information, and recording medium for performing method

Abstract

The invention relates to a method and an apparatus for providing scrap iron classification information and a recording medium for performing the method. Disclosed are a method for providing scrap iron classification information by image analysis, an apparatus for classifying scrap iron, and a recording medium for performing the method. The method includes: obtaining, by a receiving unit, a loading state image captured in a state in which a plurality of scrap iron is loaded onto a loading device; obtaining, by the processor, a segmented image including the target scrap iron from the loading state image using a segmentation model in which segmentation is performed on the target scrap iron, the target scrap iron being any one of the plurality of scrap iron; obtaining, by the processor, item information and grade information corresponding to the target scrap iron using a classification model in which classification is performed on the segmented images and analysis is performed on the classified images in units of images; and providing scrap iron classification information including the item information and the grade information by the processor.

Description

Technical Field

[0001] The technical field of the present disclosure relates to a method of providing result information of image classification of an object loaded on a loading device, a single object, etc., and to a technical field of a method for providing scrap classification information for an area including one or more scrap irons based on image analysis. Background Art

[0002] Recently, with the development of the logistics industry, loading devices are widely used to load various objects or move objects from one place to another. However, multiple objects loaded on a loading device can be managed or loaded at the position of each product or specification by a worker, but it may be difficult to check the position every time. Therefore, a segmentation method can be used, in which the entire area of the loading device on which multiple objects are loaded is divided to monitor images of each area. Generally, in the segmentation method for image analysis related to scrap iron, an optical system is easily installed at the unloading position to obtain an image without the need for hardware (H / W) engineering to set the position and angle of the optical system (closed-circuit television and mechanism), enabling artificial intelligence (AI) to perform optimally. In this case, there is a limitation in that a large amount of image collection and labeling is required because methods for classifying irregular scrap iron with few consistent features based on instance segmentation (different shapes, ordinary textures, various colors (paint, rust, etc.) depending on the usage method) and various cutting methods, bending, etc. are adopted as methods for determining the grade of scrap iron. Therefore, there is a need to provide a service in which, by providing an image analysis method and system that can solve the limitation of the analysis method that requires such a large amount of image collection, performance can be improved with only a small amount of image collection and labeling.

[0003] [Related Technical Literature]

[0004] [Patent Literature]

[0005] Korean Unexamined Patent Application Publication No. 10-2011-0078566 (published on July 7, 2011): Efficient Object Loading Location Detection System Using Digital Image Recognition Summary of the Invention

[0006] The present disclosure aims to provide a technology for improving the accuracy of image analysis and classification of one or more scrap irons included in a loading state image when providing image classification information for scrap iron, and to provide a service in which an artificial intelligence (AI) model is used to collect a minimum number of images to provide highly accurate image classification information based on the collected images.

[0007] The object of the present disclosure is not limited to the above object, and there may be other technical objects.

[0008] According to one aspect of the present disclosure, there is provided a method for classifying scrap iron by image analysis, the method including the steps of: obtaining, by a receiving unit, a loading state image captured in a state where a plurality of scrap irons are loaded onto a loading device; obtaining, by a processor, a segmented image including a target scrap iron from the loading state image using a segmentation model that performs segmentation on the target scrap iron, the target scrap iron being any one of the plurality of scrap irons; obtaining, by the processor, item information and grade information corresponding to the target scrap iron using a classification model that classifies the segmented image and analyzes the classified image on a per-image basis; and providing, by the processor, scrap iron classification information including the item information and the grade information.

[0009] The step of obtaining the item information and the grade information may include the steps of: obtaining, by the processor, a target scrap iron image representing the target scrap iron by excluding a background region from the segmented image; and classifying the target scrap iron image by the processor and obtaining the item information and the grade information.

[0010] The method may further include the steps of: obtaining, by the receiving unit, a correct image representing the target scrap iron; determining, by the processor, a percentage of an overlapping region between the correct image and the target scrap iron image; when the percentage of the overlapping region exceeds a threshold overlapping percentage, obtaining, by the processor, a scrap iron determination accuracy indicating whether the target scrap iron image is an actual image of scrap iron; and providing, by the processor, the scrap iron determination accuracy as a performance metric.

[0011] The method may further include the steps of: determining, by the processor, a target weight determined based on a region size of the target scrap iron image; determining, by the processor, a target accuracy of the target scrap iron image; and applying, by the processor, the target weight to the target accuracy and determining an accuracy for the classification model.

[0012] The method may further include the following steps: obtaining, by the receiving unit, a single segmented image captured for a single scrap iron; obtaining, by the processor, a composite image using the loading state image and the single segmented image; and applying, by the processor, the segmentation model and the classification model to the composite image and providing additional scrap iron classification information.

[0013] The step of obtaining the composite image may include the following steps: obtaining, by the processor, a single scrap iron image by excluding a background region from the single segmented image; and obtaining, by the processor, the composite image by combining the loading state image and the single scrap iron image.

[0014] The step of obtaining the composite image by combining the loading state image and the single scrap iron image may include the following steps: determining, by the processor, the number of possible combinations of the single scrap iron image and the loading state image based on the area size of the single scrap iron image; and obtaining, by the processor, the composite image based on the number of possible combinations.

[0015] The step of obtaining the loading state image may include the following steps: obtaining, by the receiving unit, a loading state image captured in an updated state by being captured at the positions of the plurality of scrap irons in the loading device loaded with a plurality of identical scrap irons; and the step of obtaining the segmented image may include the following steps: obtaining, by the processor, the segmented image including the target scrap iron from the updated loading state image using the segmentation model.

[0016] When the number of pixels included in the target scrap iron image is less than a first number, the target weight may increase proportionally to a linear function corresponding to a first slope; when the number of pixels is greater than or equal to the first number and less than a second number, the target weight may increase proportionally to an exponential function having a base larger than the first slope; when the number of pixels is greater than or equal to the second number, the target weight may increase proportionally to a linear function corresponding to a second slope smaller than the first slope; and the first slope and the second slope may be positive numbers.

[0017] The step of providing the scrap iron classification information may include the following steps: obtaining, by the processor, average weight information indicating the cumulative area and / or cumulative quantity of each item and grade of the item information and the grade information corresponding to the target scrap iron among the plurality of scrap irons; and providing, by the processor, a circular chart based on the average weight information, the circular chart showing the cumulative area ratio and / or cumulative quantity ratio of each item and grade of the target scrap iron in the loading state image.

[0018] According to another aspect of the present disclosure, there is provided an apparatus for classifying scrap iron through image analysis, the apparatus including: a receiving unit configured to obtain a loading state image captured in a state where a plurality of scrap irons are loaded onto a loading device; and a processor configured to: obtain a segmented image including a target scrap iron from the loading state image using a segmentation model that performs segmentation on the target scrap iron, where the target scrap iron is any one of the plurality of scrap irons; obtain item information and grade information corresponding to the target scrap iron using a classification model that classifies the segmented image and performs analysis on the classified image on a per-image basis; and provide scrap iron classification information including the item information and the grade information.

[0019] The processor may obtain a target scrap iron image representing the target scrap iron by excluding a background region from the segmented image, and classify the target scrap iron image and obtain the item information and the grade information.

[0020] The receiving unit may obtain a correct image representing the target scrap iron, and the processor may determine a percentage of an overlapping region between the correct image and the target scrap iron image; when the percentage of the overlapping region exceeds a threshold overlapping percentage, obtain scrap iron determination accuracy indicating whether the target scrap iron image is an actual image of the scrap iron; provide the scrap iron determination accuracy as a performance metric; determine a target weight determined according to a region size of the target scrap iron image; determine a target accuracy of the target scrap iron image; and apply the target weight to the target accuracy and determine an accuracy for the classification model.

[0021] The receiving unit may obtain a single segmented image captured for a single scrap iron, and the processor may use the loading state image and the single segmented image to obtain a composite image, and apply the segmentation model and the classification model to the composite image and provide additional scrap iron classification information.

[0022] According to another aspect of the present disclosure, there is provided a computer-readable non-transitory recording medium on which a program for implementing the method of the first aspect is recorded. Description of the Drawings

[0023] By referring to the accompanying drawings and describing in detail exemplary embodiments of the present disclosure, the above and other objects, features, and advantages of the present disclosure will become more apparent to those of ordinary skill in the art, where:

[0024] Figure 1is a block diagram schematically showing the configuration of an apparatus for classifying scrap iron through image analysis according to an embodiment of the present disclosure;

[0025] Figure 2 is a flowchart showing an operation of an apparatus for classifying scrap iron providing scrap iron classification information according to an embodiment of the present disclosure;

[0026] Figure 3 is a flowchart schematically showing an operation of an apparatus for classifying scrap iron performing image analysis on scrap iron according to layer changes according to an embodiment of the present disclosure;

[0027] Figure 4 is a diagram schematically showing an example of an apparatus for classifying scrap iron performing segmentation and then classification according to an embodiment of the present disclosure;

[0028] Figure 5 is a diagram for describing an example in which an apparatus for classifying scrap iron according to an embodiment of the present disclosure performs accuracy evaluation on a classification model;

[0029] Figure 6 is a diagram for describing an example in which an apparatus for classifying scrap iron according to an embodiment of the present disclosure obtains item information and grade information of each scrap iron based on an imaging result of a plurality of scrap irons or a single scrap iron included in a loading device;

[0030] Figure 7 is a diagram for describing an example in which an apparatus for classifying scrap iron according to an embodiment of the present disclosure enhances data based on an imaging result of a single scrap iron included in a loading device;

[0031] Figure 8 is a diagram for describing an example in which an apparatus for classifying scrap iron according to an embodiment of the present disclosure performs segmentation on a synthetic image;

[0032] Figure 9 is a diagram for describing an example in which an apparatus for classifying scrap iron according to an embodiment of the present disclosure performs segmentation on an updated loading state image; and

[0033] Figure 10 is a diagram schematically showing an example of an apparatus for classifying scrap iron providing scrap iron classification information according to an embodiment of the present disclosure. Detailed Description

[0034] Advantages and features of the present disclosure and methods for implementing the present disclosure will be clearly understood with reference to the accompanying drawings and embodiments described in detail below. However, the present disclosure is not limited to the embodiments to be disclosed below, but may be implemented in various different forms. The embodiments are provided to fully explain the present embodiments and to fully explain the scope of the present disclosure to those skilled in the art.

[0035] The terms used herein are provided only for the purpose of describing embodiments of the present disclosure and are not intended to be limiting. In this specification, unless the context clearly indicates otherwise, the singular forms include the plural forms. It should be understood that the terms "comprising" and / or "including" used herein specify the presence of some of the described components, but do not exclude the presence or addition of one or more other components. Throughout the specification, the same reference numerals denote the same components, and "and / or" includes each combination of one or more of the above components. It should be understood that although terms such as "first", "second", etc. may be used to describe various components, these components are not limited by these terms. These terms are only used to distinguish one component from another. Therefore, it should be understood that the first component to be described below may be the second component within the technical scope of the present disclosure.

[0036] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. In addition, it should be understood that terms should not be interpreted in an idealized or overly formal sense unless expressly defined herein, such as terms defined in a commonly used dictionary.

[0037] As shown in the accompanying drawings, spatial relative terms such as "below", "beneath", "lower", "above", "upper", etc. may be used to facilitate the description of the relationship between one component and other components. In addition to the directions shown in the accompanying drawings, spatial relative terms should be understood to include different directions during the use or operation of the element. For example, when the component shown in the figure is flipped, the component described as "below" or "beneath" may ultimately be placed "above" another component. Therefore, the exemplary term "below" may include both downward and upward directions. Components may be arranged in different directions, and thus spatial relative terms may be interpreted according to this arrangement.

[0038] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

[0039] Figure 1 is a block diagram schematically showing the configuration of a device 100 for classifying scrap iron by image analysis according to an embodiment of the present disclosure.

[0040] Referring to Figure 1 , the device 100 for classifying scrap iron may include a receiving unit 110 and a processor 120.

[0041] According to one embodiment, the receiving unit 110 may obtain a loading state image captured in a state where a plurality of scrap irons are loaded onto the loading device.

[0042] The processor 120 according to one embodiment may obtain a segmentation image including a target scrap iron from the loading state image using a segmentation model that performs segmentation on the target scrap iron (which is any one of the plurality of scrap irons). In addition, the processor 120 may use a classification model to obtain item information and grade information corresponding to the target scrap iron, the classification model performing classification on the segmentation image and analyzing the classified images on a per-image basis. In addition, the processor 120 may provide scrap iron classification information including the item information and the grade information.

[0043] In addition, in the process where the receiving unit 110 obtains the loading state image, and the processor 120 obtains the segmentation image by performing segmentation, obtains item information and grade information corresponding to the segmentation image by performing classification, and provides the scrap iron classification information including the item information and the grade information, the device 100 for classifying scrap iron by image analysis may be combined with various conventional networks (e.g., the Internet, a mobile communication network, etc.) and it should be noted that there is no particular limitation on the network.

[0044] In addition, those skilled in the art should understand that other general components in addition to Figure 1 those shown in may also be included in the device 100 for classifying scrap iron by image analysis. For example, the device 100 for classifying scrap iron by image analysis may also include a memory (not shown) for storing the loading state image, the segmentation image, the item information and the grade information corresponding to the segmentation image, etc., and may also include a transmission unit (not shown) for providing the scrap iron classification information or a display unit (not shown) for displaying the scrap iron classification information. Alternatively, those skilled in the art will understand that in another embodiment, some of the components shown in Figure 1 may be omitted.

[0045] The device 100 for classifying scrap iron by image analysis according to one embodiment may be used by a user, may be linked with any type of handheld wireless communication device equipped with a touch screen panel (e.g., a mobile phone, a smartphone, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet computer, etc.), and in addition, may be included in or linked with a device having a basis for installing and executing an application (e.g., a desktop personal computer (PC), a tablet computer, a laptop computer, an Internet protocol television (IPTV) including a set-top box).

[0046] The device 100 for classifying scrap iron through image analysis can be implemented as a terminal such as a computer, which performs operations through a computer program to achieve the functions described in this specification.

[0047] The device 100 for classifying scrap iron through image analysis according to one embodiment may include a system (not shown) and a related server (not shown) that provide classification information about scrap iron, but the present disclosure is not limited thereto. The server according to one embodiment may support an application that provides a service for giving classification information about scrap iron.

[0048] Hereinafter, an example in which the device 100 for classifying scrap iron through image analysis according to one embodiment independently obtains and provides classification result information according to a preset scrap iron classification method will be mainly described. However, as described above, the device 100 may perform the above functions in combination with the server. That is, the device 100 for classifying scrap iron and the server according to one embodiment may be implemented in an integrated manner in terms of their functions, and the server may be omitted. Thus, it can be seen that the present disclosure is not limited to any one embodiment.

[0049] In one embodiment, the device 100 for classifying scrap iron and the server may be linked to each other, and by performing a process for classifying scrap iron and a process for providing classification results, the configuration for providing scrap iron classification information may be executed by the server or by the device 100 for classifying scrap iron. For example, the device 100 for classifying scrap iron may operate as a server, and the device 100 for classifying scrap iron and the server are hereinafter collectively referred to as the device 100 for classifying scrap iron.

[0050] Figure 2 is a flowchart showing the operation of the device 100 for classifying scrap iron according to one embodiment of the present disclosure to provide scrap iron classification information.

[0051] Referring to operation S210, the device 100 for classifying scrap iron according to one embodiment may obtain a loading state image captured in a state where a plurality of scrap irons are loaded onto the loading device. In one embodiment, the device 100 for classifying scrap iron may obtain a loading state image captured from the upper side of the loading device, and in this case, a plurality of scrap irons are being loaded onto the loading device. Therefore, the device 100 for classifying scrap iron may obtain a loading state image including a plurality of scrap irons.

[0052] Referring to operation S220, the device 100 for classifying scrap iron according to one embodiment may obtain a segmentation image including a target scrap iron from a loading state image using a segmentation model that performs segmentation on the target scrap iron, which is any one of a plurality of scrap irons. In one embodiment, the segmentation model may include semantic segmentation and instance segmentation. In one embodiment, the device 100 for classifying scrap iron may obtain a scrap iron region corresponding to each of the plurality of scrap irons by performing segmentation on the plurality of scrap irons included in the loading state image. When the device 100 for classifying scrap iron performs segmentation using semantic segmentation, the device 100 for classifying scrap iron may separately classify the plurality of scrap irons by classifying the pixels of the loading state image into physical units. That is, the device 100 for classifying scrap iron may perform segmentation on the scrap iron in any one of the plurality of scrap iron regions in units of pixels to classify each of the plurality of scrap irons included in the loading state image into separate scrap irons. Analysis of the scrap iron region based on the result of performing semantic segmentation may be performed in units of pixels. For example, the device 100 for classifying scrap iron may obtain a quadrilateral region including at least one pixel corresponding to each of the plurality of scrap irons as a scrap iron region in units of pixels. Therefore, the device 100 for classifying scrap iron may obtain a segmentation image, which is an image corresponding to each scrap iron region obtained from the loading state image. That is, in one embodiment, each scrap iron region representing the target scrap iron (which is any one of the plurality of scrap irons) may correspond to the segmentation image. Therefore, the device 100 for classifying scrap iron may obtain a plurality of segmentation images of the target scrap iron, which is one of the plurality of scrap irons, using the segmentation model.

[0053] Referring to operation S230, the device 100 for classifying scrap iron according to one embodiment can use a classification model to obtain item information and grade information corresponding to the target scrap iron. The classification model performs classification on the segmented images and analyzes the classified images on a per-image basis. In one embodiment, the classification may be a process of analyzing or classifying scrap iron based on the feature information obtained from each image for each segmented image. The device 100 for classifying scrap iron can use the classification model to analyze the multiple segmented images obtained in operation S220 on a per-image basis. Thus, the device 100 for classifying scrap iron can obtain item information and grade information corresponding to each analyzed image. In one embodiment, the item information corresponds to the feature information about the scrap iron and may include, for example, heavy scrap iron information, light scrap iron information, etc. In addition, in one embodiment, the device 100 for classifying scrap iron can obtain grade information and item information corresponding to the target scrap iron by performing classification on the target scrap iron image. In addition, the device 100 for classifying scrap iron according to one embodiment can obtain a target scrap iron image representing the target scrap iron by excluding the background region from the segmented image. The segmented images obtained in operation S220 may be images including background regions other than the target scrap iron. In one embodiment, the background region may include the wall region of the loading device, the boundary region of the loading device, etc. Thus, the device 100 for classifying scrap iron can obtain a target scrap iron image representing the region corresponding to the target scrap iron excluding the background region, and obtain item information and grade information about the target scrap iron by performing classification on the target scrap iron image. Thus, when obtaining item information and grade information about the target scrap iron, incorrect situations regarding the analysis results that may occur according to the background region can be prevented. In addition, in one embodiment, the device 100 for classifying scrap iron can perform an evaluation process on the segmentation model. The device 100 for classifying scrap iron according to one embodiment can obtain a correct image representing the target scrap iron. In one embodiment, the correct image may be an image representing the correct answer for the scrap iron obtained from the user terminal or an external server. That is, the correct image may be an image corresponding to the actual scrap iron region, and the actual scrap iron region can be used to check whether the image corresponds to the actual scrap iron regions for multiple scrap irons obtained from the user terminal or an external server. The device 100 for classifying scrap iron can determine the percentage of the overlapping region between the correct image and the target scrap iron image. For example, the target scrap iron image may be an image in which the background region is excluded from the segmented image obtained using the segmentation model. Thus, the target scrap iron image may be the same as the correct image representing the actual scrap iron, but may be different depending on the result of performing segmentation.Therefore, the device 100 for classifying scrap iron can determine the percentage of the overlapping area between the correct image and the target scrap iron image, and when the percentage of the overlapping area exceeds the threshold overlapping percentage, the device 100 for classifying scrap iron can obtain the scrap iron determination accuracy indicating whether the target scrap iron image is the actual image of the scrap iron. In one embodiment, the threshold overlapping percentage can represent the minimum standard overlapping percentage, at which the obtained target scrap iron image can be predicted to correspond to the actual scrap iron. That is, when the percentage of the overlapping area between the correct image and the target scrap iron image exceeds the threshold overlapping percentage (e.g., 50%), the device 100 for classifying scrap iron can determine that the target scrap iron image corresponds to the actual scrap iron, and in this case, the device 100 for classifying scrap iron can obtain the scrap iron determination accuracy indicating whether the corresponding target scrap iron image is the actual image of the scrap iron. Therefore, the device 100 for classifying scrap iron can provide the scrap iron determination accuracy as a performance metric. For example, in one embodiment, the threshold overlapping percentage is the ratio that can be predicted to correspond to the actual scrap iron and can correspond to the standard for primary sorting. That is, the device 100 for classifying scrap iron can determine that the scrap iron is not scrap iron because when the percentage of the overlapping area is less than or equal to the threshold overlapping percentage, the target scrap iron image is predicted not to correspond to the scrap iron, and the device 100 for classifying scrap iron can mainly determine that the corresponding scrap iron is the predicted scrap iron because when the percentage of the overlapping area exceeds the threshold overlapping percentage, the target scrap iron image is predicted to correspond to the scrap iron. Therefore, the device 100 for classifying scrap iron can obtain the scrap iron determination accuracy indicating whether the predicted scrap iron image mainly determined to be the predicted scrap iron is the actual image of the scrap iron. For example, the device 100 for classifying scrap iron can check whether the predicted scrap iron image is an image for a complete scrap iron. That is, when the area including the wall image of the loading device, the boundary image of the loading device, etc. (which is not the scrap iron in the predicted scrap iron image) is greater than or equal to the preset percentage, or when the degree of correspondence between the predicted scrap iron image and the actual scrap iron is less than the preset percentage, the device 100 for classifying scrap iron can determine that the image is not the actual image of the scrap iron. The device 100 for classifying scrap iron can obtain the scrap iron determination accuracy indicating whether the image is an image of the scrap iron to provide a performance metric for the segmentation model. Therefore, the device 100 for classifying scrap iron can perform segmentation using the segmentation model corresponding to when the scrap iron determination accuracy is greater than or equal to the preset percentage (e.g., 90% to 95% or more). That is, the device 100 for classifying scrap iron can determine the performance of the segmentation model corresponding to when the scrap iron determination accuracy is less than the preset percentage as low performance, and thus can not apply the corresponding segmentation model as the model for performing segmentation.In another embodiment, the device 100 for classifying scrap iron can update a preset percentage corresponding to the accuracy of scrap iron determination based on the threshold overlap percentage. For example, when the threshold overlap percentage corresponds to an adjacent percentage of the preset percentage (e.g., from 48% or closer to 50% to less than 50%), the device 100 for classifying scrap iron can increase the preset percentage corresponding to the accuracy of scrap iron determination by a certain level (e.g., from 90% to 95% or more to 93% to 98% or more), and perform an evaluation on the segmentation model only for the percentage adjacent to the threshold overlap percentage. In addition, the device 100 for classifying scrap iron according to one embodiment can perform an evaluation process on the classification model. The device 100 for classifying scrap iron can determine a target weight determined according to the area size of the target scrap iron image. The device 100 for classifying scrap iron can determine the target accuracy of the target scrap iron image. In addition, the device 100 for classifying scrap iron can determine the accuracy of the classification model by applying the target weight to the target accuracy. In one embodiment, the target accuracy of the target scrap iron image can include the scrap iron determination accuracy indicating whether the image is actual scrap iron as described above, and can also include item accuracy and grade accuracy for item information and grade information corresponding to each target scrap iron obtained in operation S230. That is, the device 100 for classifying scrap iron can determine the target accuracy indicating whether the obtained item information and grade information correspond to the actual items and grades of the target scrap iron. Therefore, the device 100 for classifying scrap iron can determine the target accuracy and then update the target accuracy according to the area size of the target scrap iron image. For example, the device 100 for classifying scrap iron can determine target weights differently assigned to each of a plurality of target scrap iron images. The device 100 for classifying scrap iron can assign a higher weight to scrap iron having a large area size of the target scrap iron image. In one embodiment, since the model can be more suitable for accurately determining scrap iron with large sizes and high error rates, the device 100 for classifying scrap iron can determine the accuracy for the classification model based on the updated target accuracy by assigning a higher weight to scrap iron having a large area size of the target scrap iron image. For example, the device 100 for classifying scrap iron can determine the target weight to linearly increase in proportion to the area size of the target scrap iron image. In another embodiment, when performing semantic segmentation, the device 100 for classifying scrap iron can differently determine the degree of increase proportional to the preset area size range.For example, when the number of pixels included in the target scrap iron image is less than the first number, the target weight can increase proportionally to a linear function corresponding to the first slope. When the number of pixels is greater than or equal to the first number and less than the second number, the target weight can increase proportionally to an exponential function having a base larger than the first slope. When the number of pixels is greater than or equal to the second number, the target weight can increase proportionally to a linear function corresponding to a second slope smaller than the first slope, and the first slope and the second slope can be positive numbers. For example, the device 100 for classifying scrap iron according to an embodiment can determine the area size of the target scrap iron image based on the number of pixels. Therefore, when the number of pixels included in the target scrap iron image is less than the first number, which is a preset number, it can be determined that the target weight increases proportionally to a linear function corresponding to the first slope. In addition, when the number of pixels included in the target scrap iron image is greater than or equal to the first number and less than the second number, it can be determined that the target weight increases proportionally to an exponential function. In one embodiment, the first slope can correspond to a constant corresponding to the linear function. The device 100 for classifying scrap iron can apply an exponential function having a base larger than the constant corresponding to the first slope to the range where the number of pixels is greater than or equal to the first number and less than the second number. That is, the degree of increase in the target weight when the number of pixels is greater than or equal to the first number and less than the second number can be greater than the degree of increase in the target weight when the number of pixels is less than the first number. In addition, to determine the target weight, when the number of pixels included in the target scrap iron image is greater than or equal to the second number, the device 100 for classifying scrap iron can apply a linear function corresponding to a second slope smaller than the first slope. In one embodiment, the second slope can correspond to a constant corresponding to the linear function, or can correspond to a constant smaller than the constant corresponding to the first slope. That is, the degree of increase in the target weight when the number of pixels is greater than or equal to the second number can be less than the degree of increase in the target weight when the number of pixels is greater than or equal to the first number and less than the second number. In addition, the constants corresponding to each of the first slope, the base of the exponential function, and the second slope can be positive numbers. Therefore, the degree of increase in the target weight can be determined and applied differently according to the range of the area size including the target scrap iron image. In this case, the degree of increase in the target weight when the number of pixels is greater than or equal to the second number can be the smallest, and the degree of increase in the target weight when the number of pixels is greater than or equal to the first number and less than the second number can be the largest. Therefore, the device 100 for classifying scrap iron has the effect of more appropriately determining the accuracy of the classification model by determining the weight differently according to the area size based on the number of pixels when performing semantic segmentation. In one embodiment, the area greater than or equal to the first number and less than the second number can be the widest area, which can include the largest amount of scrap iron.In this case, the device 100 for classifying scrap iron may apply an exponential function to the degree to which the target weight increases as the importance of the size of the corresponding area is determined to be higher, and thus further apply the weight according to the area size (area). In addition, in one embodiment, a larger area size may have a greater impact on the classification accuracy. However, since the sizes of a plurality of scrap irons corresponding to the number of pixels being greater than or equal to the second number may correspond to the case where the sizes of the plurality of scrap irons are all greater than the size of the reference range, it may not be very meaningful to make a large difference between the plurality of scrap irons. In addition, since the sizes of a plurality of scrap irons corresponding to the number of pixels being less than the first number may correspond to the case where the sizes of the plurality of scrap irons are less than the size of the reference range, it may be meaningful to make the difference between the plurality of scrap irons within a range larger than when the number of pixels is greater than or equal to the second number. Therefore, the device 100 for classifying scrap iron may determine the accuracy of the classification model by determining the degree of increase in the target weight in the order of the range where the number of pixels is greater than or equal to the first number and less than the second number, the range where the number of pixels is less than the first number, and the range where the number of pixels is greater than or equal to the second number. Therefore, the device 100 for classifying scrap iron may use the classification model corresponding to when the classification model accuracy is greater than or equal to a preset percentage to obtain the accuracy for the classification model to perform classification. That is, the device 100 for classifying scrap iron may use the classification model to perform classification, and the classification model shows a high target accuracy for scrap iron with a large area size of the target scrap iron image.

[0054] Referring to operation S240, the device 100 for classifying scrap iron according to one embodiment may provide scrap iron classification information including item information and grade information. As described above, the device 100 for classifying scrap iron may provide the item information and grade information about the target scrap iron obtained using the segmentation model and the classification model as the scrap iron classification information.

[0055] Figure 3 is a flowchart schematically showing an operation of the device 100 for classifying scrap iron according to an embodiment of the present disclosure to perform image analysis on scrap iron according to layer change.

[0056] Referring to Figure 3, the device 100 for classifying scrap iron according to an embodiment can determine whether there is a loading device (loading box) and a gripper. Accordingly, the layer can be measured when the loading device exists and the gripper does not exist. In addition, image alignment can be performed only when the layer is measured and then the layer is changed, and a segmentation image can be obtained by performing separate extraction of the scrap iron through segmentation as described above in operation S220. Thereafter, item information and grade information can be obtained for each scrap iron by performing classification as described above in operation S230, and the final grade of each scrap iron can be determined.

[0057] Figure 4 FIG. is a diagram schematically showing an example in which the device 100 for classifying scrap iron according to an embodiment of the present disclosure performs segmentation and then performs classification.

[0058] Refer to Figure 4 , the device 100 for classifying scrap iron according to an embodiment can obtain a segmentation image by separately extracting a plurality of scrap irons into each scrap iron region in pixel units by using a segmentation model. In addition, the device 100 for classifying scrap iron can obtain a target scrap iron image representing the target scrap iron by excluding the background region from the segmentation image. Accordingly, as Figure 4 shown in the lower part of, classification can be performed on the target scrap iron image from which the background region has been excluded, and item information and grade information about the target scrap iron can be obtained.

[0059] Figure 5 FIG. is a diagram for describing an example in which the device 100 for classifying scrap iron according to an embodiment of the present disclosure performs accuracy evaluation on a classification model.

[0060] Refer to Figure 5 , the device 100 for classifying scrap iron according to an embodiment can assign different target weights according to the region size of the target scrap iron image. Figure 5The artificial intelligence (AI) performance measurement method shown on the left side is for the general accuracy measurement method (acc accuracy), and the same evaluation is performed because the same weight is assigned to the analysis results of large-sized scrap iron and small-sized scrap iron. In the case of the general accuracy measurement method, the equation of the number of correct answers / total number can be applied. In contrast, according to one embodiment, the area-weighted accuracy (AWA) area-weighted accuracy method can be applied. In one embodiment, the device 100 for classifying scrap iron can measure the accuracy by differently assigning target weights according to the area size (area magnitude). In the case of the AWA method, the equation of the number of pixels in the correct area / the number of pixels in the entire area can be applied. Therefore, the device 100 for classifying scrap iron can perform classification according to a model that determines the importance of large-sized scrap iron to be higher using the AWA method to determine the accuracy of the classification model.

[0061] Figure 6 FIG. is an example for describing a device 100 for classifying scrap iron according to an embodiment of the present disclosure that obtains item information and grade information of each scrap iron based on the result of imaging a plurality of scrap irons or a single scrap iron included in a loading device.

[0062] Refer to Figure 6 , according to an embodiment, the device 100 for classifying scrap iron can obtain a loading state image captured in a state where a plurality of scrap irons are loaded, as described above in operation S210, or can obtain a single segmentation image captured at a plurality of angles for a single scrap iron, as Figure 6 shown in the lower part of. Therefore, the device 100 for classifying scrap iron can obtain a segmentation image by separately extracting scrap iron from the loading state image using a segmentation model, and can obtain item information and grade information corresponding to the target scrap iron by classifying the segmentation image using a classification model by separately extracting scrap iron from the loading state image. In addition, when the device 100 for classifying scrap iron obtains a single segmentation image, the device 100 for classifying scrap iron can obtain item information and grade information corresponding to the single scrap iron by classifying the single segmentation image using a classification model. Therefore, item matching information matching the image and its item information can be obtained for each scrap iron and stored in a database.

[0063] Figure 7 FIG. is an example for describing a device 100 for classifying scrap iron according to an embodiment of the present disclosure that enhances data based on the result of imaging one scrap iron or a single scrap iron included in a loading device.

[0064] Refer to Figure 7, according to one embodiment, the device 100 for classifying scrap iron can obtain item information and grade information corresponding to each target scrap iron by extracting a target scrap iron from among a plurality of scrap irons loaded onto the loading device and performing classification separately, and store the obtained item information and grade information in a database. In addition, the device 100 for classifying scrap iron can perform segmentation and classification on newly obtained images by classifying the loading state image, rather than directly classifying a single segmented image captured for a single scrap iron. Therefore, the device 100 for classifying scrap iron can also store multiple pieces of obtained information in the database. In this regard, this will be described with reference to Figure 8 as follows.

[0065] Figure 8 FIG. is a diagram for describing an example in which the device 100 for classifying scrap iron according to an embodiment of the present disclosure performs segmentation on a composite image.

[0066] Referring to Figure 8 , according to one embodiment, the device 100 for classifying scrap iron can obtain a composite image using the loading state image and a single segmented image. The device 100 for classifying scrap iron can obtain a single scrap iron image by excluding the background region from the single segmented image. In addition, the device 100 for classifying scrap iron can obtain a composite image by combining the loading state image and the single scrap iron image. As Figure 8As shown in [reference], a single scrap iron corresponding to the shaded area can be combined into the loading state image. Thus, the device 100 for classifying scrap iron can perform segmentation based on the newly obtained composite image by accumulating the single segmented images on the loading state image. The device 100 for classifying scrap iron can provide additional scrap iron classification information including item information and grade information corresponding to the target scrap iron obtained by applying the segmentation model and the classification model to the composite image. In another embodiment, the device 100 for classifying scrap iron can determine the number of possible combinations of single scrap iron to be synthesized into the loading state image based on the area size of the single scrap iron image. In addition, the device 100 for classifying scrap iron can obtain the composite image based on the number of possible combinations. For example, the device 100 for classifying scrap iron according to one embodiment can first obtain a composite image in which the single scrap iron image is combined with the loading state image, and then obtain a composite image in which multiple single scrap iron images are combined. For example, when the ratio of the number of pixels included in the single scrap iron image area to the total number of pixels in the cross-sectional area corresponding to the loading device is less than the first percentage (e.g., 10%), the device 100 for classifying scrap iron can determine the number of possible combinations such that for each area where the horizontal length of the loading device is divided by the first value (e.g., five), the single scrap iron image can be combined into two or more areas. For example, the device 100 for classifying scrap iron can be configured to allow the single scrap iron images to overlap and combine in the first area (e.g., five areas) obtained by dividing the horizontal length of the loading device by the first value. That is, in this case, the device 100 for classifying scrap iron can obtain multiple composite images by determining the number of possible combinations as one of two to five. In addition, when the ratio of the number of pixels included in the single scrap iron image area to the total number of pixels in the cross-sectional area corresponding to the loading device is greater than or equal to the first percentage and less than the second percentage (e.g., 20%), the device 100 for classifying scrap iron can determine the number of possible combinations such that for each area where the horizontal length of the loading device is divided by the second value smaller than the first value (e.g., three), the single scrap iron image can be combined into two or more areas. For example, the device 100 for classifying scrap iron can be configured to allow the single scrap iron images to overlap and combine in the second area (e.g., three areas) obtained by dividing the horizontal length of the loading device by the second value. That is, in this case, the device 100 for classifying scrap iron can obtain multiple composite images by determining the number of possible combinations as one of two or three.In addition, when the ratio of the number of pixels included in a single scrap iron image region to the total number of pixels in the cross-sectional area corresponding to the loading device is greater than or equal to a second percentage, the device 100 for classifying scrap iron can determine the number of possible combinations such that for each region where the horizontal length of the loading device is divided by a third value (e.g., two) smaller than a second value, a single scrap iron image can be combined into two regions. For example, the device 100 for classifying scrap iron can be configured to allow a single scrap iron image to overlap and combine in a third region (e.g., two regions) obtained by dividing the horizontal length of the loading device by the third value. That is, in this case, the device 100 for classifying scrap iron can also obtain a composite image by determining the number of possible combinations to be two. Therefore, the device 100 for classifying scrap iron can perform more data augmentation using the above data augmentation process.

[0067] Figure 9 FIG. is an example for describing where the device 100 for classifying scrap iron according to an embodiment of the present disclosure performs segmentation on an updated loading state image.

[0068] Refer to Figure 9 FIG., the device 100 for classifying scrap iron according to an embodiment can obtain a loading state image that is updated by being captured in a state where the positions of multiple scrap irons are updated in a loading device on which multiple identical scrap irons are loaded. That is, the device 100 for classifying scrap iron can obtain an updated loading state image newly captured in a state where the positions of multiple scrap irons are changed by a gripper. In addition, the device 100 for classifying scrap iron can obtain a segmented image including the target scrap iron from the updated loading state image using a segmentation model. Refer to Figure 9 FIG., the device 100 for classifying scrap iron can obtain a loading state image that is updated by being captured in a state where the positions of multiple scrap irons are updated by a gripper, and can obtain an updated loading state image in which the positions of multiple scrap irons are updated by dividing the regions of the loading state image and updating the order of the corresponding image regions differently. The device 100 for classifying scrap iron can perform segmentation on the updated loading state image by newly obtaining an image in which the positions of multiple scrap irons are updated using a loading device on which multiple identical scrap irons are loaded. Therefore, the device 100 for classifying scrap iron can perform more data augmentation.

[0069] Figure 10 FIG. is a diagram schematically showing an example in which the device 100 for classifying scrap iron according to an embodiment of the present disclosure provides scrap iron classification information.

[0070] Refer to Figure 10, according to one embodiment, the device 100 for classifying scrap iron can obtain average weight information, which indicates the cumulative area and / or cumulative quantity of the items and grades corresponding to each of the item information and grade information of the target scrap iron among a plurality of scrap irons. In addition, the device 100 for classifying scrap iron can provide a pie chart based on the average weight information, and the pie chart shows the cumulative area ratio and / or cumulative quantity ratio of each item and grade of the target scrap iron in the loading state image. The device 100 for classifying scrap iron according to one embodiment can not only provide the grade information of each target scrap iron, but also provide the item information. For example, the device 100 for classifying scrap iron can provide information about heavy scrap iron, light scrap iron, etc. (which indicates the items of scrap iron), and in addition, can also provide information about grade A, grade B, etc. (which indicates the grades of scrap iron). The device 100 for classifying scrap iron can accumulate all the determination results of the entire loading device and finally calculate the area (or quantity). In addition, the device 100 for classifying scrap iron can not only measure the area ratio, but also measure the weight of each grade by tabulating the average weight information of each area of each grade / item. According to one embodiment, different from the traditional related technologies, the device 100 for classifying scrap iron can not only provide grade information, but also provide item information. Therefore, the device 100 for classifying scrap iron has the effect of being convenient to be applied to countries (by country or by steel manufacturer) with the same items but different grades.

[0071] According to one embodiment, by performing image analysis and image classification using a segmentation model and a classification model, a high-performance scrap iron classification process can be provided based on a small number of collected images. In addition, when providing scrap iron classification information, there is an advantage that since the processes of segmentation and classification are performed separately, effective data augmentation can be performed on the scrap iron images, and since the scrap iron image and image classification information are obtained by performing evaluation processing on the segmentation model and the classification model, the accuracy of the classification result can be improved.

[0072] Various embodiments of the present disclosure may be implemented as software including one or more instructions stored in a storage medium (e.g., a memory) readable by a machine (e.g., a display device or a computer). For example, a processor (e.g., processor 120) of the machine may call at least one stored instruction from the storage medium and execute the instruction. This enables the device to operate according to the at least one called instruction to perform at least one function. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The storage medium readable by the device may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where data is temporarily stored in the storage medium.

[0073] According to one embodiment, a method according to various embodiments disclosed in the present disclosure may be included in and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., by downloading or uploading) through an application store (e.g., Play Store TM), or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium (e.g., the memory of a manufacturer's server, an application store's server, or an intermediate server).

[0074] According to an embodiment of the present disclosure, by performing image analysis and image classification using a segmentation model and a classification model, a high-performance scrap iron classification process can be provided based on a small number of collected images.

[0075] In addition, when providing scrap iron classification information, there is an advantage in that effective data augmentation can be performed on the scrap iron images because the processes for segmentation and classification are performed separately.

[0076] In addition, since the scrap iron images and image classification information are obtained by performing an evaluation process on the segmentation model and the classification model, the accuracy of the classification result can be improved.

[0077] The effects of the present disclosure are not limited to the above effects, and those skilled in the art can clearly understand other effects not described from the above detailed description.

[0078] Although the present disclosure has been described with reference to the accompanying drawings, the present disclosure is not limited to the disclosed embodiments and drawings, and those skilled in the art will understand that various changes can be made in form and detail without departing from the spirit and scope of the present disclosure. Therefore, the disclosed method should be described from an exemplary rather than a restrictive perspective. Even when an embodiment is described and the effects of the configuration according to the present disclosure are not explicitly described, the effects predictable by the configuration can be recognized. The scope of the present disclosure is not defined by the detailed description of the present disclosure, but by the appended claims, and includes all modifications and equivalents falling within the scope of the appended claims, and will be interpreted as being included in the present disclosure.

[0079] Cross - reference to related applications

[0080] This application claims the priority and benefit of Korean Patent Application No. 10 - 2023 - 0188832, filed on December 21, 2023, the entire contents of which are incorporated herein by reference in their entirety.

Claims

1. A method for providing scrap metal classification information by image analysis, the method for providing scrap metal classification information comprising the following steps: obtaining, by a receiving unit, a loading state image captured when a plurality of scrap irons are loaded on a loading device; obtaining, by a processor, a segmented image including target scrap from the loading state image using a segmentation model, the segmentation model performing segmentation on the target scrap, the target scrap being any one of the plurality of scraps; obtaining, by the processor, item information and grade information corresponding to the target scrap iron using a classification model that performs classification on the segmented images and performs analysis on the classified images in units of images; and The processor provides scrap metal classification information including the item information and the grade information.

2. The method for providing scrap iron classification information according to claim 1, wherein: The step of obtaining the project information and the level information comprises the following steps: obtaining, by the processor, a target scrap iron image representing the target scrap iron by excluding a background area from the segmented image; and The processor performs classification on the target scrap iron image and obtains the item information and the grade information.

3. The method for providing scrap iron classification information according to claim 2, further comprising the following steps: The receiving unit obtains a correct image representing the target scrap iron; determining, by the processor, a percentage of overlapping areas between the correct image and the target scrap metal image; when the percentage of the overlapped area exceeds a threshold overlapped percentage, obtaining, by the processor, a scrap determination accuracy indicating whether the target scrap image is an actual image of scrap; as well as The scrap determination accuracy is provided by the processor as a performance indicator.

4. The method for providing scrap iron classification information according to claim 2, further comprising the following steps: Determining, by the processor, a target weight determined according to the area size of the target scrap iron image; determining, by the processor, a target accuracy of the target scrap metal image; as well as The target weight is applied by the processor to the target accuracy and an accuracy for the classification model is determined.

5. The method for providing scrap iron classification information according to claim 1, further comprising the following steps: The receiving unit obtains a single segmented image captured for a single scrap iron; obtaining, by the processor, a composite image using the loading state image and the single segmented image; as well as The segmentation model and the classification model are applied by the processor to the composite image and provide additional scrap classification information.

6. The method for providing scrap iron classification information according to claim 5, wherein: The step of obtaining the composite image comprises the following steps: obtaining, by the processor, a single scrap iron image by excluding a background region from the single segmented image; and The composite image is obtained by the processor by combining the loading state image with the single scrap iron image.

7. The method for providing scrap iron classification information according to claim 6, wherein: The step of obtaining the composite image by combining the loading state image with the single scrap iron image comprises the following steps: determining, by the processor, the number of possible combinations of the single scrap image and the loading state image based on the area size of the single scrap image; and The composite image is obtained, by the processor, based on the number of possible combinations.

8. The method for providing scrap iron classification information according to claim 1, wherein: The step of obtaining the loading state image includes the steps of: obtaining, by the receiving unit, a loading state image updated by being captured in a state updated in the loading device loaded with a plurality of identical scraps at the positions of the plurality of scraps; and The step of obtaining the segmented image includes the step of obtaining, by the processor, the segmented image including the target scrap from the updated loading state image using the segmented model.

9. The method for providing scrap iron classification information according to claim 4, wherein: When the number of pixels included in the target scrap iron image is less than a first number, the target weight is increased in proportion to a linear function corresponding to a first slope; When the number of pixels is greater than or equal to the first number and less than a second number, the target weight increases in proportion to an exponential function having a base greater than the first slope; When the number of pixels is greater than or equal to the second number, the target weight increases in proportion to a linear function corresponding to a second slope smaller than the first slope; and The first slope and the second slope have positive values.

10. The method for providing scrap iron classification information according to claim 1, wherein: The step of providing the scrap iron classification information comprises the following steps: obtaining, by the processor, average weight information indicating a cumulative area and / or cumulative quantity of each item and grade of the item information and the grade information corresponding to the target scrap among the plurality of scraps; and A circular graph showing a cumulative area ratio and / or a cumulative quantity ratio of each item and grade of the target scrap iron in the loading state image is provided by the processor based on the average weight information.

11. A device for providing scrap iron classification information by image analysis, the device for providing scrap iron classification information is used to classify scrap iron, the device for providing scrap iron classification information comprises: a receiving unit configured to obtain a loading state image captured in a state where a plurality of scrap irons are loaded on the loading device; as well as A processor configured to: obtaining a segmented image including target scrap iron from the loading state image using a segmentation model, the segmentation model performing segmentation on the target scrap iron, the target scrap iron being any one of the plurality of scrap irons; obtaining item information and grade information corresponding to the target scrap iron using a classification model that performs classification on the segmented images and performs analysis on the classified images in units of images; and Scrap classification information including the item information and the grade information is provided.

12. The device for providing scrap iron classification information according to claim 11, wherein: The processor is configured to: A target scrap image representing the target scrap is obtained by excluding a background area from the segmented image; and classification is performed on the target scrap image and the item information and the grade information are obtained.

13. The device for providing scrap iron classification information according to claim 12, wherein: The receiving unit obtains a correct image representing the target scrap iron, and, The processor is configured to: Determining the percentage of overlapping areas of the correct image and the target scrap metal image; when the percentage of the overlapping area exceeds a threshold overlapping percentage, obtaining a scrap determination accuracy indicating whether the target scrap image is an actual image of scrap; providing the scrap iron determination accuracy as a performance indicator; Determining a target weight according to the area size of the target scrap iron image; determining a target accuracy of the target scrap metal image; and The target weight is applied to the target accuracy and an accuracy for the classification model is determined.

14. The device for providing scrap iron classification information according to claim 11, wherein: The receiving unit obtains a single segmented image captured for a single scrap iron, and The processor is configured to: obtaining a composite image using the loading state image and the single segmented image; and The segmentation model and the classification model are applied to the composite image and provide additional scrap classification information.

15. A recording medium for executing the method for providing scrap iron classification information according to any one of claims 1 to 10, wherein the recording medium is a computer-readable recording medium on which a program for executing the method for providing scrap iron classification information on a computer is recorded.

Images