Iron scrap classification method and iron scrap classification device through image analysis

Abstract

To provide iron scrap classification information by image analysis for a region including one or more iron scraps.SOLUTION: There are provided a method which includes the steps of: allowing a reception part to acquire a loaded state image imaged in a state in which a plurality of iron scraps is loaded on a loading facility; allowing a processor to acquire a divided image including an object iron scrap which is one of the plurality of iron scraps by a loaded state image using a segmentation model performing segmentation for the object iron scrap; allowing a processor to acquire item information and grade information corresponding to the object iron scrap using a classification model performing classification for the divided image, and performing analysis by image unit; and providing iron scrap classification information including the item information and the grade information, and an iron scrap classification device and a recording medium for performing the method.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The technical field of the present disclosure relates to a method for providing image classification result information for an article loaded on a loading equipment or a single article, etc., and relates to the technical field of a method for providing iron scrap classification information by image analysis for an area including one or more iron scraps.

Background Art

[0002] Due to the recent increase in the logistics industry, in fact, loading equipment for loading various articles or for loading articles to move them from one place to another is widely used. However, although a plurality of articles loaded on the loading equipment can be managed or loaded according to the operator's judgment by product type and specification position, it may be difficult to check this every time. Therefore, a segmentation method may be used to divide the entire area of the loading equipment on which a plurality of articles are loaded so that the images of each area can be monitored. Generally, in the case of a segmentation method for iron scrap-related image analysis, the optical system (CCTV and mechanism parts) can be easily installed at the loading (unloading) location without H / W engineering to set the position and angle of the optical system so that AI can exhibit optimal performance, and images are acquired. In such a case, there are few consistent features (various shapes by utilization methods, plain textures, various color senses (painting, rust, etc.)), and an irregular iron scrap due to various cutting methods, warping, etc. is classified by instance segmentation to determine the grade of the iron scrap. Therefore, in fact, there is a limitation that a large amount of image collection and labeling are required. Therefore, it is necessary to provide an image analysis method and system that can solve the limitation of such an analysis method that requires a large amount of image collection, so as to provide a service that can improve performance with only a smaller amount of image collection and labeling.

Prior Art Documents

Patent Documents

[0003] Korean Patent Publication No. 10-2011-0078566 (July 7, 2011) Efficient Article Loading Position Detection System Using Digital Video Recognition

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present disclosure is to improve the accuracy of image analysis and classification for one or more iron scraps included in the loading state image in providing image classification information for iron scraps, and to collect a minimum amount of images using an AI model and provide a service for providing highly accurate image classification information associated therewith.

[0005] The problem to be solved by the present disclosure is not limited to the above technical problems, and there may be other technical problems.

Means for Solving the Problems

[0006] As a technical means for achieving the above-described technical problems, a method for classifying iron scraps through image analysis according to a first aspect of the present disclosure includes: a receiving unit obtaining a loading state image captured in a state where a plurality of iron scraps are loaded on loading equipment; a processor performing segmentation on a target iron scrap that is any one of the plurality of iron scraps and obtaining a segmented image including the target iron scrap in the loading state image using a segmentation model; the processor performing classification on the segmented image and obtaining item information and grade information corresponding to the target iron scrap using a classification model that performs analysis on an image-by-image basis; and the processor providing iron scrap classification information including the item information and the grade information.

[0007] In addition, the step of obtaining the item information and the grade information may include: the step in which the processor obtains an image of the target iron scrap showing the target iron scrap excluding the background area in the divided image; and the step in which the processor performs classification on the image of the target iron scrap to obtain the item information and the grade information.

[0008] In addition, it may further include: the step in which the receiving unit obtains a correct image showing the target iron scrap; the step in which the processor determines the ratio of the overlapping area between the correct image and the image of the target iron scrap; the step in which the processor obtains the accuracy of iron scrap determination indicating whether the image of the target iron scrap is actually an image of the iron scrap when the ratio of the overlapping area exceeds the critical overlapping ratio; and the step in which the processor provides the accuracy of iron scrap determination as a performance index.

[0009] In addition, it may further include: the step in which the processor determines a target weighting value determined by the size of the area of the image of the target iron scrap; the step in which the processor determines the target accuracy for the image of the target iron scrap; and the step in which the processor applies the target weighting value to the target accuracy to determine the accuracy for the classification model.

[0010] In addition, it may further include: the step in which the receiving unit obtains a single divided image captured for a single iron scrap; the step in which the processor obtains a composite image using the loaded state image and the single divided image; and the step in which the processor applies the segmentation model and the classification model to the composite image to provide additional iron scrap classification information.

[0011] Also, the step of obtaining the composite image may include: the processor obtaining a single iron scrap image excluding the background area from the single divided image; and the processor combining the loaded state image and the single iron scrap image to obtain the composite image.

[0012] Also, the step of combining the loaded state image and the single iron scrap image to obtain the composite image may include: the processor determining the number of combinable single iron scraps to be synthesized into the loaded state image based on the size of the single iron scrap image area; and the processor obtaining the composite image based on the number of combinable ones.

[0013] Also, the step of obtaining the loaded state image may include: the receiving unit obtaining an updated loaded state image captured in the loading equipment where the same plurality of iron scraps are loaded and the positions of the plurality of iron scraps are updated; and the step of obtaining the divided image may include: the processor using the segmentation model to obtain a divided image including the target iron scrap from the updated loaded state image.

[0014] Also, when the number of pixels included in the target iron scrap image is less than a first number, the target weight value increases proportionally to a linear function corresponding to a first slope; when the number of pixels is greater than or equal to the first number and less than a second number, the target weight value increases proportionally to an exponential function having a bottom larger than the first slope; when the number of pixels is greater than or equal to the second number, the target weight value increases proportionally to a linear function corresponding to a second slope showing a slope smaller than the first slope, and the first slope and the second slope can be positive numbers.

[0015] In addition, the step of providing the iron scrap classification information includes the step in which the processor obtains average weight information indicating the cumulative area and / or cumulative number for each item and grade with respect to the item information and the grade information corresponding to the target iron scrap among the plurality of iron scraps; and the step in which the processor provides a circular graph indicating the cumulative area ratio and / or cumulative number ratio for each item and grade with respect to the target iron scrap in the loaded state image based on the average weight information.

[0016] An iron scrap classification apparatus through image analysis according to a second aspect of the present disclosure can include a receiving unit that obtains a loaded state image captured in a state where a plurality of iron scraps are loaded on a loading equipment; and a processor that obtains item information and grade information corresponding to the target iron scrap by using a segmentation model that performs segmentation on a target iron scrap that is any one of the plurality of iron scraps to obtain a divided image including the target iron scrap in the loaded state image, and performing classification on the divided image and performing analysis in image units by using a classification model, and provides iron scrap classification information including the item information and the grade information.

[0017] In addition, the processor can obtain a target iron scrap image showing the target iron scrap excluding the background area in the divided image, and perform the classification on the target iron scrap image to obtain the item information and the grade information.

[0018] In addition, the receiving unit acquires a correct image indicating the target iron scrap, the processor determines a ratio of an overlapping area between the correct image and the target iron scrap image, and when the ratio of the overlapping area exceeds a critical overlapping ratio, acquires an iron scrap determination accuracy indicating whether the target iron scrap image is actually an image of iron scrap, provides the iron scrap determination accuracy as a performance index, determines a target weighted value determined by the size of the area of the target iron scrap image, determines a target accuracy for the target iron scrap image, and can determine the accuracy for the classification model by applying the target weighted value to the target accuracy.

[0019] In addition, the receiving unit acquires a single divided image captured for a single iron scrap, and the processor can acquire a composite image using the loaded state image and the single divided image, and provide additional iron scrap classification information by applying the segmentation model and the classification model to the composite image.

[0020] According to a third aspect of the present disclosure, a computer-readable non-transitory recording medium on which a program for implementing the method of the first aspect is recorded can be provided.

Advantages of the Invention

[0021] According to an embodiment of the present disclosure, there is an effect that a high-performance iron scrap classification process can be provided by collecting a small number of images in terms of performing image analysis and image classification using a segmentation model and a classification model.

[0022] In addition, in providing iron scrap classification information, there is an advantage that effective data augmentation for iron scrap images is possible in that the processes of segmentation and classification are performed separately.

[0023] In addition, in terms of performing an evaluation process for the segmentation model and the classification model to obtain an image of iron scrap and obtain image classification information, there is an effect that the accuracy of the classification result can be improved.

[0024] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

Brief Description of Drawings

[0025]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0026] Advantages, features, and methods for achieving them in the present disclosure will become clear by referring to the embodiments described in detail below together with the attached drawings. However, the present disclosure is not limited to the embodiments disclosed below and can be embodied in various different forms. However, these embodiments are provided to make the disclosure complete and to fully inform those of ordinary skill in the relevant technical field of the scope of the present disclosure.

[0027] The terms used in this specification are for explaining the embodiments and are not intended to limit the present disclosure. In this specification, the singular form includes the plural form unless otherwise specifically stated in the context. The terms "comprises" and / or "comprising" used in the specification do not exclude the presence or addition of one or more other components in addition to the recited components. The same reference numerals throughout the drawings refer to the same components, and "and / or" includes each and all combinations of the recited components. Although terms such as "first", "second", etc. are used to describe various components, it goes without saying that these components are not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it goes without saying that the first component referred to below may be the second component within the technical idea of the present disclosure.

[0028] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in a sense commonly understood by one of ordinary skill in the relevant art. Also, terms defined in commonly used dictionaries will not be interpreted ideally or overly unless specifically defined otherwise.

[0029] Spatially relative terms such as “below,” “beneath,” “lower,” “above,” “upper,” etc. may be used to easily describe the correlation between one component and another as illustrated in the drawings. Spatially relative terms should be understood to include different directions of components relative to each other during use or operation in addition to the directions illustrated in the drawings. For example, when the components illustrated in the drawings are turned over, a component described as “below” or “beneath” another component may be placed “above” the other component. Thus, the exemplary term “below” can include all directions of below and above. The component can be oriented in other directions as well, and accordingly, the spatially relative terms can be interpreted according to the orientation.

[0030] Hereinafter, embodiments will be described in detail with reference to the drawings.

[0031] FIG. 1 is a block diagram schematically illustrating the configuration of an iron scrap classification apparatus 100 through image analysis according to an embodiment of the present disclosure.

[0032] Referring to FIG. 1, the iron scrap classification apparatus 100 can include a receiving unit 110 and a processor 120.

[0033] According to an embodiment, the receiving unit 110 can acquire a loaded state image captured in a state where a plurality of iron scraps are loaded on the loading equipment.

[0034] The processor 120 according to one embodiment can obtain a divided image including the target iron scrap in the loaded state image by using a segmentation model that performs segmentation on the target iron scrap, which is any one of a plurality of iron scraps. Further, the processor 120 can obtain item information and grade information corresponding to the target iron scrap by using a classification model that performs classification on the divided image and performs analysis on an image-by-image basis. Further, the processor 120 can provide iron scrap classification information including the item information and the grade information.

[0035] In addition, the iron scrap classification apparatus 100 through image analysis can obtain a loaded state image by the receiving unit 110, perform segmentation by the processor 120 to obtain a divided image, perform classification to obtain item information and grade information corresponding to the divided image, and provide iron scrap classification information including the item information and the grade information. It should be noted that there are no special restrictions on this, and it can be coupled by various conventional network combinations such as the Internet or a mobile communication network in the process.

[0036] In addition to this, those with ordinary knowledge in the relevant technical field can understand that other general components can be further included in the iron scrap classification apparatus 100 through image analysis in addition to the components illustrated in FIG. 1. For example, the iron scrap classification apparatus 100 through image analysis can further include a memory (not shown) for storing a loaded state image, a divided image, item information and grade information corresponding to the divided image, etc., and can further include a transmission unit (not shown) for providing iron scrap classification information or a display unit (not shown) for displaying iron scrap classification information. Or when following other embodiments, those with ordinary knowledge in the relevant technical field can understand that some of the components illustrated in FIG. 1 can be omitted.

[0037] The iron scrap classification device 100 through image analysis according to an embodiment can be used by a user and can be linked with all types of handheld wireless communication devices equipped with a touch screen panel, such as a mobile phone, smartphone, PDA (Personal Digital Assistant), PMP (Portable Multimedia Player), tablet PC, etc. In addition, it can also be included in or linked with devices provided with a base on which an application can be installed and executed, such as a desktop PC, tablet PC, laptop PC, and IPTV including a set-top box.

[0038] The iron scrap classification device 100 through image analysis can be implemented on a terminal such as a computer that operates through a computer program for implementing the functions described in this specification.

[0039] The iron scrap classification device 100 through image analysis according to an embodiment can include, but is not limited to, a system (not shown) and a related server (not shown) that provide classification information for iron scrap. A server according to an embodiment can assist an application that provides a service for providing classification information for iron scrap.

[0040] Hereinafter, an embodiment in which the iron scrap classification device 100 through image analysis according to an embodiment independently obtains and provides classification information results by a preset iron scrap classification method will be mainly described. However, as described above, it may also be performed through linkage with a server. That is, it can be seen that the iron scrap classification device 100 through image analysis according to an embodiment and the server can be integrally implemented in terms of their functions, the server may be omitted, and it is not limited to any one embodiment.

[0041] In one embodiment, the iron scrap classification device 100 and the server can be interlocked, and the configuration for providing iron scrap classification information by performing the iron scrap classification process and the classification result information providing process can be performed by the server or may be performed by the iron scrap classification device 100. For example, the iron scrap classification device 100 can operate as a server, and hereinafter, it will be uniformly described with the iron scrap classification device 100.

[0042] FIG. 2 is a flowchart showing each stage in which the iron scrap classification device 100 according to an embodiment of the present disclosure provides iron scrap classification information.

[0043] Referring to step S210, the iron scrap classification device 100 according to an embodiment can acquire a loaded state image that is imaged in a state where a plurality of iron scraps are loaded on the loading equipment. In one embodiment, the iron scrap classification device 100 can acquire a loaded state image that is imaged from the upper surface of the loading equipment downward, and at this time, the loading equipment may be in a state where a plurality of iron scraps are loaded. Therefore, the iron scrap classification device 100 can acquire a loaded state image including a plurality of iron scraps.

[0044] Referring to stage S220, the iron scrap classification device 100 according to an embodiment can obtain a divided image including a target iron scrap in a loaded state image by using a segmentation model that performs segmentation on a target iron scrap that is any one of a plurality of iron scraps. In one embodiment, the segmentation model can include semantic segmentation and instance segmentation. In one embodiment, the iron scrap classification device 100 can obtain scrap regions corresponding to each of the plurality of iron scraps by performing segmentation on the plurality of iron scraps included in the loaded state image. When the iron scrap classification device 100 performs segmentation using semantic segmentation, the plurality of iron scraps can be individually classified by classifying the pixels of the loaded state image in physical units. That is, the iron scrap classification device 100 can perform segmentation on the iron scrap in any one of the plurality of iron scrap regions in pixel units in order to classify each of the plurality of iron scraps included in the loaded state image into individual iron scraps. The analysis of the scrap region based on the result of performing semantic segmentation can be performed in pixel units. For example, the iron scrap classification device 100 can obtain a rectangular region including at least one or more pixels corresponding to each of the plurality of iron scraps as a scrap region based on pixel units. Therefore, the iron scrap classification device 100 can obtain a divided image that is an image corresponding to each scrap region obtained in the loaded state image. That is, in one embodiment, each scrap region indicating a target iron scrap that is any one of the plurality of iron scraps can correspond to the divided image. Therefore, the iron scrap classification device 100 can obtain a plurality of divided images for a target iron scrap that is one of the plurality of iron scraps by using the segmentation model.

[0045] Referring to step S230, the iron scrap classification device 100 according to an embodiment can obtain item information and grade information corresponding to the target iron scrap by performing classification on the divided images and using a classification model to perform analysis on an image-by-image basis. In one embodiment, the classification can be a process of analyzing or classifying iron scrap based on the feature information obtained from each image for each divided image. The iron scrap classification device 100 can perform analysis on an image-by-image basis for each of the plurality of divided images obtained in step S220 by using the classification model. Therefore, the iron scrap classification device 100 can obtain item information and grade information corresponding to each analysis image. In one embodiment, the item information corresponds to the characteristic information of the iron scrap and can include, for example, weight information, lightweight information, and the like. Also, in one embodiment, the iron scrap classification device 100 can perform classification on the target iron scrap image and obtain grade information together with the item information corresponding to the target iron scrap. Further, the iron scrap classification device 100 according to an embodiment can obtain a target iron scrap image showing the target iron scrap excluding the background area in the divided image. The divided image obtained in step S220 can be an image including a background area other than the target iron scrap. In one embodiment, the background area can include the wall area of the loading equipment, the edge area of the loading equipment, and the like. Therefore, the iron scrap classification device 100 can obtain a target iron scrap image showing the area corresponding to the target iron scrap excluding the background area, perform classification on the target iron scrap image, and obtain item information and grade information for the target iron scrap. Therefore, in obtaining the item information and grade information for the target iron scrap, it is possible to prevent an error situation for the analysis result that may be caused by the background area. Also, in one embodiment, the iron scrap classification device 100 can perform an evaluation process on the segmentation model. The iron scrap classification device 100 according to an embodiment can obtain a ground truth image showing the target iron scrap.In one embodiment, the correct image may mean an image showing the correct answer for the iron scrap acquired from the user terminal or an external server. That is, the correct image may be an image corresponding to the actual iron scrap area that can confirm whether it corresponds to the actual iron scrap area acquired from the user terminal or an external server for a plurality of iron scraps. The iron scrap classification device 100 can determine the ratio of the overlapping area between the correct image and the target iron scrap image. For example, the target iron scrap image may be a segmented image acquired using a segmentation model and excluding the background area. Therefore, the target iron scrap image may be the same as the correct image showing the actual iron scrap, or may be different depending on the segmentation result. Therefore, the iron scrap classification device 100 can determine the ratio of the overlapping area between the correct image and the target iron scrap image. When the ratio of the overlapping area exceeds the critical overlapping ratio, the iron scrap determination accuracy indicating whether the target iron scrap image is actually an image of the iron scrap can be obtained. In one embodiment, the critical overlapping ratio can indicate the minimum reference overlapping ratio that can predict that the acquired target iron scrap image corresponds to the actual iron scrap. That is, when the ratio of the overlapping area between the correct image and the target iron scrap image of the iron scrap classification device 100 exceeds the critical overlapping ratio (for example, 50 percent), it can be determined that the target iron scrap image corresponds to the actual iron scrap. In this case, the iron scrap determination accuracy regarding whether the corresponding target iron scrap image is actually the iron scrap can be obtained. Therefore, the iron scrap classification device 100 can provide the iron scrap determination accuracy as a performance index. For example, in one embodiment, the critical overlapping ratio is a ratio that can be predicted to correspond to the actual iron scrap and may correspond to a criterion for primary selection.That is, when the ratio of the overlapping area is equal to or less than the critical overlapping ratio, the iron scrap classifier 100 can determine that it does not correspond to iron scrap by predicting that it does not correspond to iron scrap. When the ratio of the overlapping area exceeds the critical overlapping ratio, the corresponding iron scrap can be primarily determined as predicted iron scrap by predicting that it corresponds to iron scrap. Therefore, it is possible to obtain the iron scrap determination accuracy indicating whether the predicted iron scrap image, which is primarily determined as predicted iron scrap, is actually an image of iron scrap. For example, the iron scrap classifier 100 can perform a confirmation as to whether the predicted iron scrap image is an image of a complete iron scrap. That is, when the area including the wall image of the loading equipment that is not iron scrap and the edge image of the loading equipment in the predicted iron scrap image is equal to or more than a preset percentage, or when the degree to which the predicted iron scrap image corresponds to the actual iron scrap is less than the preset percentage, it can be determined that it is not actually an image of iron scrap. The iron scrap classifier 100 can provide a performance index for the segmentation model by obtaining the iron scrap determination accuracy indicating whether it is an image of iron scrap. Therefore, when the iron scrap determination accuracy is equal to or more than a preset percentage (for example, 90% to 95% or more), the iron scrap classifier 100 can perform segmentation using the corresponding segmentation model. That is, when the iron scrap determination accuracy is less than the preset percentage, the iron scrap classifier 100 cannot be applied as a model for performing segmentation by determining that the performance of the corresponding segmentation model is inferior performance. In other embodiments, the iron scrap classifier 100 may update the preset percentage corresponding to the iron scrap determination accuracy according to the critical overlapping ratio.For example, when the critical overlap ratio of the iron scrap classification device 100 corresponds to a percentage adjacent to the preset percentage (for example, when it is 48% or more and less than 50% adjacent to 50%), the preset percentage corresponding to the iron scrap determination accuracy is increased by a certain level (for example, increased from 90% to 95% or more to 93% to 98% or more increased by a certain level). Thus, the evaluation of the segmentation model may be performed as long as it corresponds to the percentage adjacent to the critical overlap ratio. Also, the iron scrap classification device 100 according to an embodiment can perform an evaluation process for the classification model. The iron scrap classification device 100 can determine a target weighted value determined by the size of the region of the target iron scrap image. The iron scrap classification device 100 can determine the target accuracy for the target iron scrap image. Also, the iron scrap classification device 100 can apply the target weighted value to the target accuracy to determine the accuracy for the classification model. In one embodiment, the target accuracy for the target iron scrap image can include, as described above, the iron scrap determination accuracy regarding whether it is actual iron scrap, and can further include the item accuracy and the grade accuracy regarding the item information and the grade information corresponding to each target iron scrap obtained in step S230. That is, the iron scrap classification device 100 can determine the target accuracy indicating whether the obtained item information and grade information correspond to the actual item and grade for the target iron scrap. Therefore, after determining the target accuracy, the iron scrap classification device 100 can update the target accuracy according to the size of the target iron scrap image region. For example, the iron scrap classification device 100 can determine different target weighted values for each of a plurality of target iron scrap images. The iron scrap classification device 100 can assign a higher weighted value to the iron scrap with a larger target iron scrap image region size.In one embodiment, in terms of being a model that is more suitable for accurately determining large-sized iron scrap with a high actual yield rate, the iron scrap classification device 100 can determine the accuracy of the classification model based on the target accuracy that is updated by assigning a high weight value to large-sized iron scrap in the area of the target iron scrap image. For example, the iron scrap classification device 100 can determine the target weight value so as to increase linearly proportionally to the size of the target iron scrap image area. In other embodiments, when the iron scrap classification device 100 performs semantic segmentation, it may be determined to increase differently in proportion based on the size range of the preset area. For example, when the number of pixels included in the target iron scrap image is less than the first number, the target weight value increases in proportion to a linear function corresponding to the first slope, and when the number of pixels is greater than or equal to the first number and less than the second number, the target weight value increases in proportion to an exponential function having a base larger than the first slope, and when the number of pixels is greater than or equal to the second number, the target weight value can increase in proportion to a linear function corresponding to a second slope showing a slope smaller than the first slope, and the first slope and the second slope can be positive numbers. For example, the iron scrap classification device 100 according to one embodiment can determine the size of the area of the target iron scrap image based on the number of pixels. Therefore, when the number of pixels included in the target iron scrap image is less than the first number which is the preset number, the target weight value can be determined to increase in proportion to a linear function corresponding to the first slope. Also, when the number of pixels included in the target iron scrap image is greater than or equal to the first number and less than the second number, the target weight value can be determined to increase in proportion to an exponential function. In one embodiment, the first slope can correspond to a constant corresponding to the linear function. The iron scrap classification device 100 can apply an exponential function having a base larger than the constant corresponding to the first slope to the range where the number of pixels is greater than or equal to the first number and less than the second number. That is, the degree to which the target weight value increases when the number of pixels is greater than or equal to the first number and less than the second number may be larger than the degree to which the target weight value increases when the number of pixels is less than the first number.In addition, when determining the target weighted value, the iron scrap classification device 100 can be configured such that a linear function corresponding to a second slope smaller than the first slope is applied when the number of pixels included in the target iron scrap image is equal to or greater than a second number. In one embodiment, the second slope can correspond to a constant corresponding to the linear function and can be a constant smaller than the constant corresponding to the first slope. That is, when the number of pixels is equal to or greater than the second number, the degree of increase in the target weighted value can be made smaller again than the degree of increase in the target weighted value when the number of pixels is equal to or greater than the first number and less than the second number. Also, the constants corresponding to the first slope, the base of the exponential function, and the second slope can be positive numbers. Therefore, it is possible to determine and apply the degree of increase in the target weighted value to be different depending on the range including the size of the area of the target iron scrap image. According to such a case, the degree of increase in the target weighted value can be minimized when the number of pixels is equal to or greater than the second number, and the degree of increase in the target weighted value can be maximized when the number of pixels is equal to or greater than the first number and less than the second number. Therefore, when the iron scrap classification device 100 performs semantic segmentation, there is an effect that the accuracy of the classification model can be determined more appropriately by determining different weighted values according to the size of the area based on the number of pixels. In one embodiment, the first... The area with more than the first number and less than the second number is the area with the widest range, and can be the range that may contain the most iron scraps. In this case, the iron scrap classification device 100 can be configured such that a weight value based on the size (area) of the area is further applied by making the importance for the size of the corresponding area be judged even higher, so that an exponential function is applied to increase the target weight value. Also, in one embodiment, the larger the area size, the more it can further affect the classification accuracy. However, since it is possible to handle the case where all the sizes of a plurality of iron scraps corresponding to the case where the number of pixels is equal to or more than the second number are large, it may not be very meaningful to provide a large difference between them. Also, since it is possible to handle the case where the sizes of a plurality of iron scraps corresponding to the case where the number of pixels is less than the first number are smaller than the size of the reference range, it may be meaningful to provide a larger difference between them within the corresponding range than in the case where the number of pixels is equal to or more than the second number. Therefore, the iron scrap classification device 100 can determine the accuracy for the classification model by determining the degree of increase in the target weight value in the order of the range where the number of pixels is equal to or more than the first number and less than the second number, the range where the number of pixels is less than the first number, and the range where the number of pixels is equal to or more than the second number. Therefore, the iron scrap classification device 100 can perform classification using a classification model corresponding to the case where the classification model accuracy is equal to or more than a preset percentage by obtaining the accuracy for the classification model. That is, the iron scrap classification device 100 can perform classification using a classification model in which the target accuracy for iron scraps with a large area size of the target iron scrap image is shown to be high.

[0046] Referring to step S240, the iron scrap classification device 100 according to an embodiment can provide iron scrap classification information including item information and grade information. As described above, the iron scrap classification device 100 can provide item information and grade information for the target iron scrap obtained by using the segmentation model and the classification model as the iron scrap classification information.

[0047] FIG. 3 is a flowchart schematically showing each step in which the iron scrap classification device 100 according to an embodiment of the present disclosure performs image analysis on iron scrap by changing the layer.

[0048] Referring to FIG. 3, the iron scrap classification device 100 according to an embodiment can determine the presence or absence of a loading equipment (loading platform) and a grapple. Therefore, when the loading equipment is present and the grapple is absent, the layer can be measured. Also, image alignment can be performed only when the layer change is performed after measuring the layer. As described in step S220, segmentation can be performed to obtain a divided image by individually extracting iron scrap dogs. Thereafter, as described in step S230, classification can be performed to obtain item information and grade information for each iron scrap, and the final grade of each iron scrap can be determined.

[0049] FIG. 4 is a drawing schematically showing an example in which the iron scrap classification device 100 according to an embodiment of the present disclosure performs classification after performing segmentation.

[0050] Referring to FIG. 4, the iron scrap classification device 100 according to an embodiment can obtain a segmented image by individually extracting a plurality of iron scraps into respective scrap regions based on pixels using a segmentation model. Further, the iron scrap classification device 100 can obtain an object iron scrap image showing the target iron scrap excluding the background region from the segmented image. Accordingly, as shown at the lower end of FIG. 4, classification can be performed on the object iron scrap image from which the background region has been excluded to obtain item information and grade information for the target iron scrap.

[0051] FIG. 5 is a drawing for explaining an example in which the iron scrap classification device 100 according to an embodiment of the present disclosure performs an accuracy evaluation on a classification model.

[0052] Referring to FIG. 5, the iron scrap classification device 100 according to an embodiment can assign different object weighting values according to the size of the region of the object iron scrap image. The AI performance accuracy measurement method shown on the left side of FIG. 5 relates to a general accuracy measurement method (Acc-Accuracy), and equal evaluation is performed by assigning the same weighting value to the analysis results for iron scraps with a large size and iron scraps with a small size. In the case of a general accuracy measurement method, the formula of the number of correct answers / the total number can be applied. In contrast, according to an embodiment, the AWA (Area-Weighted Accuracy) area weighting value accuracy method can be applied. In one embodiment, the iron scrap classification device 100 can measure the accuracy by differentially assigning object weighting values according to the size (area size) of the region. In the case of the area weighting value accuracy method, the formula of the number of correct answer area pixels / the total area pixels can be applied. Therefore, since the iron scrap classification device 100 determines the accuracy using the area weighting value accuracy method in determining the accuracy of the classification model, classification can be performed by a model that determines that the importance of large-sized iron scraps is high.

[0053] FIG. 6 is a drawing for explaining an example of obtaining item information and grade information of each iron scrap based on imaging results of a plurality of iron scraps or a single iron scrap included in a loading equipment by an iron scrap classification apparatus 100 according to an embodiment of the present disclosure.

[0054] Referring to FIG. 6, as described in step S210, the iron scrap classification apparatus 100 according to an embodiment may obtain a loaded state image captured in a state where a plurality of iron scraps are loaded, and as shown at the lower end of FIG. 6, may obtain a single divided image captured at multiple angles with respect to a single iron scrap. Therefore, the iron scrap classification apparatus 100 may individually extract iron scraps from the loaded state image by performing segmentation using a segmentation model to obtain a divided image, and perform classification on the divided image using a classification model to obtain item information and grade information corresponding to the target iron scrap. Further, when the iron scrap classification apparatus 100 obtains a single divided image, it can obtain item information and grade information corresponding to the single iron scrap by performing classification on the single divided image using a classification model. Therefore, item matching information in which the image and item information for each iron scrap are matched can be obtained and stored in a database.

[0055] FIG. 7 is a drawing for explaining an example of enhancing data based on imaging results of one iron scrap or a single iron scrap included in a loading equipment by an iron scrap classification apparatus 100 according to an embodiment of the present disclosure.

[0056] Referring to FIG. 7, an iron scrap classification apparatus 100 according to an embodiment can extract a target iron scrap that is any one of a plurality of iron scraps loaded on a loading equipment, perform individual classification, obtain item information and grade information corresponding to each target iron scrap, and store them in a database. Further, the iron scrap classification apparatus 100 does not immediately perform classification on a single divided image captured for a single iron scrap, but can perform segmentation and classification on a newly acquired image by applying it to a loaded state image. Accordingly, a plurality of acquired information can be further stored in the database. In this regard, it can be described with reference to FIG. 8.

[0057] FIG. 8 is a drawing for explaining an example in which an iron scrap classification apparatus 100 according to an embodiment of the present disclosure performs segmentation on a composite image.

[0058] Referring to FIG. 8, a scrap iron sorting apparatus 100 according to an embodiment can obtain a composite image by using a loaded state image and a single split image. The scrap iron sorting apparatus 100 can obtain a single scrap iron image excluding the background area in the single split image. Further, the scrap iron sorting apparatus 100 can combine the loaded state image and the single scrap iron image to obtain a composite image. As shown in FIG. 8, a single scrap iron corresponding to the hatched area can be combined with the loaded state image. Therefore, the scrap iron sorting apparatus 100 can perform segmentation based on the newly obtained composite image such that the single split image is accumulated in the loaded state image. The scrap iron sorting apparatus 100 can provide additional scrap iron sorting information including item information and grade information corresponding to the target scrap iron obtained by applying a segmentation model and a classification model to the composite image. In another embodiment, the scrap iron sorting apparatus 100 can determine the number of single scrap irons that can be combined with the loaded state image based on the size of the single scrap iron image area. Further, the scrap iron sorting apparatus 100 can obtain a composite image based on the number of single scrap irons that can be combined. For example, the scrap iron sorting apparatus 100 according to an embodiment can preferentially obtain a composite image in which one single scrap iron image is combined with the loaded state image, and may further obtain a composite image in which a plurality of single scrap iron images are combined later. For example, when the ratio of the number of pixels included in the single scrap iron image area to the total number of pixels in the cross-sectional area corresponding to the loading equipment is less than the first percentage (for example, 10%), the scrap iron sorting apparatus 100 can determine the number of single scrap irons that can be combined such that the single scrap iron image can be combined with two or more areas for each area obtained by dividing the horizontal length of the loading equipment by the first value (for example, 5). For example, the scrap iron sorting apparatus 100 can cause the single scrap iron image to be overlapped and combined in the first area (for example, 5 areas) obtained by dividing the horizontal length of the loading equipment by the first value.That is, in this case, the iron scrap classification device 100 can obtain a plurality of composite images by determining the number of composites to be one of two to five. Further, when the ratio of the number of pixels included in the single iron scrap image area to the total number of pixels in the cross-sectional area corresponding to the loading equipment is not less than the first percentage and less than the second percentage (for example, 20%), the iron scrap classification device 100 divides the horizontal length of the loading equipment by a second value smaller than the first value (for example, 3), and can determine the number of composites so that a single iron scrap image can be combined into two or more areas for each area. For example, the iron scrap classification device 100 can overlap and combine a single iron scrap image in a second area (for example, three areas) obtained by dividing the horizontal length of the loading equipment by the second value. That is, in this case, the iron scrap classification device 100 can obtain a plurality of composite images by determining the number of composites to be two or three. Further, when the ratio of the number of pixels included in the single iron scrap image area to the total number of pixels in the cross-sectional area corresponding to the loading equipment is not less than the second percentage, the iron scrap classification device 100 divides the horizontal length of the loading equipment by a third value smaller than the second value (for example, 2), and can determine the number of composites so that a single iron scrap image can be combined into two areas for each area. For example, the iron scrap classification device 100 can overlap and combine a single iron scrap image in a third area (for example, two areas) obtained by dividing the horizontal length of the loading equipment by the third value. That is, in this case, the iron scrap classification device 100 can further obtain one composite image by determining the number of composites to be two. Therefore, the iron scrap classification device 100 can perform more data augmentation by using the data augmentation process described above.

[0059] FIG. 9 is a drawing for explaining an example in which the iron scrap classification device 100 according to an embodiment of the present disclosure performs segmentation on an updated loading state image.

[0060] Referring to FIG. 9, an iron scrap classification device 100 according to an embodiment is a loading equipment on which a plurality of identical iron scraps are loaded, and can obtain an updated loading state image that is imaged with the positions of the plurality of iron scraps updated. That is, the iron scrap classification device 100 can obtain an updated loading state image that is newly imaged with the positions of the plurality of iron scraps changed through a grapple. Further, the iron scrap classification device 100 can obtain a segmented image including a target iron scrap in the updated loading state image by using a segmentation model. Referring to FIG. 9, the iron scrap classification device 100 may obtain an updated loading state image that is imaged with the positions of the plurality of iron scraps updated through a grapple, or may obtain an updated loading state image in which the positions of the plurality of iron scraps are updated by dividing the area of the loading state image and updating the order of each image area differently. The iron scrap classification device 100 can perform segmentation on the updated loading state image by newly obtaining an image with the positions of the plurality of iron scraps updated by using a loading equipment on which a plurality of identical iron scraps are loaded, and accordingly, can perform more data augmentation.

[0061] FIG. 10 is a drawing schematically showing an example in which an iron scrap classification device 100 according to an embodiment of the present disclosure provides iron scrap classification information.

[0062] Referring to FIG. 10, a scrap iron sorting device 100 according to an embodiment can obtain average weight information indicating the cumulative area and / or cumulative number for each item and grade with respect to the item information and grade information corresponding to the target scrap iron among a plurality of scrap irons. Further, the scrap iron sorting device 100 can provide a circular graph showing the cumulative area ratio and / or cumulative number ratio for each item and grade with respect to the target scrap iron in the loaded state image based on the average weight information. The scrap iron sorting device 100 according to an embodiment can provide not only grade information but also item information for each target scrap iron. For example, the scrap iron sorting device 100 can provide information on the weight, lightweight, etc., which indicate the items for scrap iron other than the A grade, B grade, etc., which indicate the grades for scrap iron. The scrap iron sorting device 100 can accumulate all the results determined for the entire loading equipment and finally calculate them in terms of area (or number). Further, if the scrap iron sorting device 100 tabulates the average weight information by area for each grade / item, it can measure not only the area ratio but also the total weight by grade. According to an embodiment, different from the existing prior art, since it acquires and provides not only grade information but also item information, there is an effect that it can be easily applied to countries (by country or by steelmaking company) that have the same item but different grades.

[0063] According to an embodiment, there is an effect that a high-performance scrap iron sorting process can be provided by collecting a small number of images in terms of performing image analysis and image classification using a segmentation model and a classification model. Also, in providing scrap iron sorting information, there is an advantage that effective data augmentation for scrap iron images is possible in that the processes of segmentation and classification are performed separately, and there is an effect that the accuracy of the classification result can be improved in that an evaluation process for the segmentation model and the classification model is performed to obtain an image for the scrap iron and obtain image classification information.

[0064] Various embodiments of the present disclosure may be embodied in software that includes one or more instructions stored in a storage medium (e.g., a memory) readable by a machine (e.g., a display device or a computer). For example, a processor of the machine (e.g., processor 220) can call and execute at least one of the one or more instructions stored from the storage medium. This enables the machine to be operated to perform at least one function by the at least one called instruction. The one or more instructions can include code generated by a compiler or code executable by an interpreter. The storage medium readable by the machine can be provided in the form of a non-transitory storage medium. Here, "non-transitory" only means that the storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), and this term does not distinguish between cases where data is stored semi-permanently and temporarily in the storage medium.

[0065] According to one embodiment, the methods according to various embodiments disclosed in the present disclosure may be provided included in a computer program product. The computer program product can be traded as a commodity between a seller and a purchaser. The computer program product can be distributed in the form of a storage medium readable by a machine (e.g., a compact disc read only memory (CD-ROM)), or through an application store (e.g., the Play Store™), or directly online (e.g., downloaded or uploaded) between two user devices (e.g., smartphones). In the case of online distribution, at least a part of the computer program product can be at least temporarily stored or temporarily generated in a storage medium readable by a machine, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0066] The present invention has been described with reference to the drawings illustrated, but it is not limited by the disclosed embodiments and the drawings. Those having ordinary knowledge in the technical field related to the present embodiment will understand that it can be embodied in a modified form without departing from the essential characteristics of the above description. Therefore, the disclosed method should be considered from an explanatory perspective rather than a limiting perspective. Even if the effects related to the configuration of the present invention are not explicitly described when explaining the embodiments, the effects predictable by the corresponding configuration may also be recognized. The scope of the present invention is shown not in the above description but in the claims, and all differences within the equivalent scope should be construed as being included in the present invention.

Explanation of Reference Numerals

[0067] 100: Iron Scrap Classification Device 110: Receiver 120: Processor

Claims

1. In a method for providing iron scrap classification information through image analysis, a step of a receiving unit obtaining a loaded state image captured in a state where a plurality of iron scraps are loaded on a loading equipment; a step of a processor obtaining a segmented image including the target iron scrap in the loaded state image by using a segmentation model that performs segmentation on a target iron scrap that is any one of the plurality of iron scraps; a step of the processor obtaining item information and grade information corresponding to the target iron scrap by using a classification model that performs classification on the segmented image and performs analysis on an image-by-image basis; and a step of the processor providing iron scrap classification information including the item information and the grade information; A method comprising:

2. The step of obtaining the item information and the grade information is a step of the processor obtaining a target iron scrap image showing the target iron scrap excluding a background area in the segmented image; and a step of the processor performing the classification on the target iron scrap image to obtain the item information and the grade information; The method according to claim 1, comprising:

3. a step of the receiving unit obtaining a correct image showing the target iron scrap; a step of the processor determining a ratio of an overlapping area between the correct image and the target iron scrap image; a step of the processor obtaining an iron scrap determination accuracy indicating whether the target iron scrap image is actually an image of an iron scrap when the ratio of the overlapping area exceeds a critical overlap ratio; and a step of the processor providing the iron scrap determination accuracy as a performance index; The method according to claim 2, further comprising:

4. a step of the processor determining a target weighting value determined by the size of the area of the target iron scrap image; a step of the processor determining a target accuracy for the target iron scrap image; and a step of the processor applying the target weighting value to the target accuracy to determine an accuracy for the classification model; The method according to claim 2, further comprising:

5. a step of the receiving unit obtaining a single segmented image captured for a single iron scrap; The step of the processor obtaining a composite image by using the loaded state image and the single split image; and The step of the processor applying the segmentation model and the classification model to the composite image to provide additional iron scrap classification information; The method according to claim 1, further comprising.

6. The step of obtaining the composite image is The step of the processor obtaining a single iron scrap image excluding the background area from the single split image; and The step of the processor combining the loaded state image and the single iron scrap image to obtain the composite image; The method according to claim 5, comprising.

7. The step of combining the loaded state image and the single iron scrap image to obtain the composite image is The step of the processor determining the number of synthesizable single iron scraps to be synthesized into the loaded state image based on the size of the single iron scrap image area; and The step of the processor obtaining the composite image based on the number of synthesizable ones; The method according to claim 6, comprising.

8. The step of obtaining the loaded state image is The step of the receiving unit obtaining an updated loaded state image that is imaged in the loaded equipment on which the same plurality of iron scraps are loaded with the positions of the plurality of iron scraps updated; The step of obtaining the split image is The step of the processor obtaining a split image including the target iron scrap in the updated loaded state image by using the segmentation model; The method according to claim 1, comprising.

9. When the number of pixels included in the target iron scrap image is less than a first number, the target weighting value increases in proportion to a linear function corresponding to a first slope, When the number of pixels is equal to or greater than the first number and less than a second number, the target weighting value increases in proportion to an exponential function having a bottom larger than the first slope, When the number of pixels is equal to or greater than the second number, the target weighting value increases in proportion to a linear function corresponding to a second slope showing a slope smaller than the first slope, The first slope and the second slope are positive numbers; The method according to claim 4.

10. The step of providing the iron scrap classification information is The step of the processor obtaining average weight information indicating the cumulative area and / or cumulative number for each item and grade corresponding to the target iron scrap among the plurality of iron scraps; and The step of the processor providing a circular graph indicating the cumulative area ratio and / or cumulative number ratio for each item and grade for the target iron scrap in the loaded state image based on the average weight information; The method according to claim 1, comprising:

11. In an iron scrap classification device that provides iron scrap classification information through image analysis, A receiving unit that obtains a loaded state image captured in a state where a plurality of iron scraps are loaded on a loading equipment; and Using a segmentation model that performs segmentation on a target iron scrap that is any one of the plurality of iron scraps, obtaining a segmented image including the target iron scrap in the loaded state image, Performing classification on the segmented image and using a classification model that performs analysis on an image-by-image basis to obtain item information and grade information corresponding to the target iron scrap, A processor that provides iron scrap classification information including the item information and the grade information; An iron scrap classification device, comprising:

12. The processor Obtaining a target iron scrap image showing the target iron scrap excluding the background area in the segmented image, Performing the classification on the target iron scrap image to obtain the item information and the grade information; The iron scrap classification device according to claim 11.

13. The receiving unit Obtaining a ground truth image showing the target iron scrap, The processor Determining the ratio of the overlapping area between the ground truth image and the target iron scrap image, When the ratio of the overlapping area exceeds a critical overlap ratio, obtaining an iron scrap determination accuracy indicating whether the target iron scrap image is actually an image of an iron scrap, Providing the iron scrap determination accuracy as a performance indicator, Determining a target weighting value determined by the size of the area of the target iron scrap image, Determining a target accuracy for the target iron scrap image, The iron scrap classification device according to claim 12, which determines the accuracy for the classification model by applying the target weight value to the target accuracy.

14. The receiving unit acquires a single segmented image captured for a single iron scrap, The processor acquires a composite image using the loaded state image and the single segmented image, The iron scrap classification device according to claim 11, which applies the segmentation model and the classification model to the composite image to provide additional iron scrap classification information.

15. A computer-readable recording medium recording a program for causing a computer to execute the method according to any one of claims 1 to 10.

Images