Method for providing iron scrap classification information by loading process and iron scrap classification device

Abstract

To improve accuracy of image analysis and classification for one or more iron scraps included in a loaded state image, in a loading process.SOLUTION: A method for providing iron scrap classification information by a loading process includes: allowing a reception part 110 to acquire a loaded state image imaged in a loading process for a plurality of iron scraps loaded on a loaded facility; allowing a processor 120 to acquire layer information determined by the height of the plurality of iron scraps and determine a region of interest, with progress of the loading process; acquiring a divided image for an object iron scrap included in the region of interest; acquiring item information and grade information for the object iron scrap; and providing iron scrap classification information including the item information and the grade information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The technical field of the present disclosure relates to a method for providing image classification result information in a cargo handling process such as an article loaded on a loading equipment or a single article, and more particularly to a technical field of a method for providing iron scrap classification information by image analysis for an area where one or more iron scraps are unloaded.

Background Art

[0002] Recently, due to the increase in the logistics industry, it is a fact that loading equipment for loading various articles or loading articles to move from one place to another is widely used. However, although a plurality of articles loaded on the loading equipment can be managed or loaded according to the type of product and the position by specification by the judgment of the operator, it may be difficult to check this every time. Therefore, a segmentation method may be used to monitor the images of each area by dividing the entire area of the loading equipment on which a plurality of articles are loaded. Generally, in the case of a segmentation method for iron scrap-related image analysis, the optical system (CCTV and mechanical parts) is easily installed at the cargo handling (unloading) location without H / W engineering to set the position and angle of the optical system so that AI can exhibit optimal performance, and images are acquired. In such a case, there are few consistent features (various shapes depending on the utilization method, plain textures, various color sensations (painting, rust, etc.)), and an irregular iron scrap due to various cutting methods, warping, etc. is classified by instance segmentation to determine the grade of the iron scrap. Therefore, there is a limitation that a large amount of image collection and labeling are required. Therefore, it is necessary to provide an image analysis method and system that can solve the limitations of such an analysis method that requires a large amount of image collection, so as to provide a service that can improve performance with only a smaller amount of image collection and labeling.

Prior Art Documents

Patent Documents

[0003] [Patent Document 1] Korean Patent Publication No. 10-2011-0078566 (July 7, 2011): Efficient article loading position detection system using digital video recognition [Summary of the Invention] [Problems to be Solved by the Invention]

[0004] The problem to be solved by the present disclosure is to improve the accuracy of image analysis and classification for one or more iron scraps included in a loading state image when providing classification information to iron scraps during the cargo handling process. It is for obtaining a divided image of a region determined as an unloading region from a loading state image captured during the cargo handling process and providing a service for providing highly accurate image classification information for the divided image.

[0005] The problem to be solved by the present disclosure is not limited to the above technical problems, and there may be other technical problems. [Means for Solving the Problems]

[0006] As a technical means for achieving the above-described technical problem, a method for providing iron scrap classification information based on a handling process according to a first aspect of the present disclosure is a method for providing iron scrap classification information based on a handling process. The method includes: a step of a receiving unit acquiring a loading state image captured during a handling process for a plurality of iron scraps loaded on loading equipment; a step of a processor acquiring layer information updated as the handling process progresses and determined by the height of the plurality of iron scraps; a step of the processor determining a region of interest based on the loading state image updated as the handling process progresses; a step of the processor acquiring a divided image for a target iron scrap that is one of the plurality of iron scraps and included in the region of interest; a step of the processor acquiring item information and grade information for the target iron scrap based on the divided image and the layer information; and a step of the processor providing iron scrap classification information including the item information and the grade information.

[0007] In addition, the method for acquiring the layer information includes a monotype using one camera or a stereo type using two or more cameras. The monotype can utilize depth information acquired through an analysis of changes in the wall surface of the loading platform obtained from an image acquired by the one camera.

[0008] In addition, in the step of determining the region of interest, the processor can determine the region of interest based on a comparison result between a first loading state image corresponding to a first time point and a second loading state image corresponding to a second time point that is temporally subsequent to the first time point.

[0009] In addition, in the step of determining the region of interest, the processor can determine the region of interest based on a difference region between the first loading state image and the second loading state image and an operation region of a grapple used in the handling process.

[0010] In addition, the step in which the processor obtains the segmented image is to obtain the segmented image including the target iron scrap from the loaded state image using a segmentation model, and the step in which the processor obtains the item information and the grade information can obtain the item information and the grade information corresponding to the target iron scrap using a classification model that performs analysis in image units.

[0011] In addition, the step of obtaining the segmented image includes: the step in which the processor performs instance segmentation on the loaded state image to obtain the segmented image for a single object; and the step in which the processor performs semantic segmentation on the loaded state image to obtain the segmented image for a clustered object.

[0012] In addition, the step of obtaining the segmented image for the clustered object can be performed on the area of the loaded state image from which the area corresponding to the single object has been excluded.

[0013] In addition, the stereotype can utilize the depth information obtained through analysis of the angular changes obtained from the images obtained for the same area from the two or more cameras located on the same plane.

[0014] In addition, the second time point is determined based on whether the grapple is included in any partial area of the vertical direction area of the loading equipment after the first time point, and the step of determining the region of interest can be performed by the processor based on the operating state of the grapple as to whether the grapple includes at least one or more iron scraps when the grapple is included in the partial area for a preset time or more.

[0015] In addition, the method for obtaining the layer information is such that the processor obtains a first layer change time point at which the height of the plurality of iron scraps changes to be equal to or greater than a critical length based on the depth information obtained by the processor according to the stereotype, and the processor obtains a second layer change time point at which the volume of the plurality of iron scraps changes to be equal to or greater than a critical percentage based on the average volume of the loading equipment, and the processor can obtain the layer information based on at least one of the first layer change time point and the second layer change time point.

[0016] The scrap iron classification device for providing scrap iron classification information by a handling process according to the second aspect of the present disclosure may include a receiving unit that obtains a loading state image captured during a handling process for a plurality of scrap irons loaded on loading equipment; and a processor that is updated as the handling process progresses, obtains layer information determined by the height of the plurality of scrap irons, determines an area of interest based on the loading state image that is updated as the handling process progresses, obtains a divided image for a target scrap iron that is one of the plurality of scrap irons and is included in the area of interest, obtains item information and grade information for the target scrap iron based on the divided image and the layer information, and provides scrap iron classification information including the item information and the grade information.

[0017] In addition, the method for obtaining the layer information includes a monotype using one camera or a stereo type using two or more cameras. The monotype can use depth information obtained through analysis of changes in the wall surface of the loading platform obtained from an image obtained by the one camera.

[0018] In addition, the processor can determine the area of interest based on a comparison result between a first loading state image corresponding to a first time point and a second loading state image corresponding to a second time point that is temporally subsequent to the first time point.

[0019] In addition, the processor can determine the region of interest based on the difference region between the first loaded state image and the second loaded state image and the operation region of the grapple used in the handling process.

[0020] According to a third aspect of the present disclosure, a computer-readable non-transitory recording medium on which a program for implementing the method of the first aspect is recorded can be provided.

Advantages of the Invention

[0021] According to an embodiment of the present disclosure, there is an effect that the accuracy of iron scrap classification can be improved by obtaining divided images for single objects and cluster objects, respectively, in terms of performing image analysis and image classification using a segmentation model and a classification model.

[0022] In addition, in terms of obtaining layer information based on monotype and stereotype, there is an effect that the accuracy of image classification and analysis for the unloading area can be increased.

[0023] In addition, in terms of determining the region of interest based on the grapple operation region, there is an effect that the accuracy of detecting the unloading area can be increased.

[0024] In addition, in terms of obtaining a divided image by using semantic segmentation and instance segmentation in parallel, there is an advantage that it is possible to complement regions where each segmentation is not accurately performed.

[0025] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

Brief Description of the Drawings

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Best Mode for Carrying Out the Invention

[0027] Advantages, features, and the ways to achieve them in the present disclosure will become clear by referring to the embodiments described in detail below together with the attached drawings. However, the present disclosure is not limited to the embodiments disclosed below and can be embodied in various different forms. Nevertheless, these embodiments are provided to make the disclosure complete and to fully inform those of ordinary skill in the relevant technical field of the scope of the present disclosure.

[0028] The terms used in this specification are for the purpose of describing the embodiments and are not intended to limit the present disclosure. In this specification, the singular form also includes the plural form unless otherwise specifically stated in the text. The terms "comprises" and / or "comprising" used in the specification do not exclude the presence or addition of one or more other components in addition to the recited components. The same reference numerals throughout the specification refer to the same components, and "and / or" includes each and all combinations of the recited components. Although terms such as "first", "second", etc. are used to describe various components, it goes without saying that these components are not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it goes without saying that the first component referred to below may be the second component within the technical idea of the present disclosure.

[0029] Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used in a meaning commonly understood by those of ordinary skill in the relevant technical field. Also, terms defined in commonly used dictionaries are not ideally or overly interpreted unless specifically defined otherwise.

[0030] Spatially relative terms such as "below", "beneath", "lower", "above", "upper", etc. can be used to easily describe the correlation between one component and another as shown in the drawings. Spatially relative terms should be understood as terms including different directions of components during use or operation in addition to the directions shown in the drawings. For example, when the components shown in the drawings are turned over, a component described as "below" or "beneath" another component can be placed "above" the other component. Therefore, the exemplary term "below" can include all directions of below and above. The components can be oriented in other directions, and accordingly, the spatially relative terms can be interpreted according to the orientation.

[0031] Hereinafter, embodiments will be described in detail with reference to the drawings.

[0032] FIG. 1 is a block diagram schematically illustrating the configuration of an iron scrap classification device 100 through image analysis according to an embodiment of the present disclosure.

[0033] Referring to FIG. 1, the iron scrap classification device 100 can include a receiving unit 110 and a processor 120.

[0034] According to an embodiment, the receiving unit 110 can obtain a loading state image captured during the handling process of a plurality of iron scraps loaded on the loading equipment.

[0035] The processor 120 according to one embodiment can obtain layer information determined by the height of a plurality of iron scraps as the handling process progresses. Further, the processor 120 can determine a region of interest based on a loading state image that is updated as the handling process progresses. Further, the processor 120 can obtain a divided image of a target iron scrap that is included in the region of interest and is one of the plurality of iron scraps. Further, the processor 120 can obtain item information and grade information for the target iron scrap based on the divided image and the layer information. Further, the processor 120 can provide iron scrap classification information including the item information and the grade information.

[0036] Further, it should be noted that the iron scrap classification device 100 that provides iron scrap classification information based on the handling process can be coupled by various conventional network combinations such as the Internet or a mobile communication network in the process of obtaining a loading state image by the receiving unit 110, obtaining layer information by the height of a plurality of iron scraps by the processor 120, determining a region of interest based on the loading state image to obtain a divided image of the iron scrap, and providing iron scrap classification information for the divided image, and there is no special restriction on this.

[0037] In addition to this, those with ordinary knowledge in the relevant technical field can understand that in addition to the components illustrated in FIG. 1, other general-purpose components may be further included in the iron scrap classification device 100 that provides iron scrap classification information through the handling process. For example, the iron scrap classification device 100 that provides iron scrap classification information through the handling process may further include a memory (not shown) that stores a loading state image, layer information, region of interest, divided image, item information and grade information corresponding to the divided image, etc., and may further include a transmission unit (not shown) that provides iron scrap classification information or a display unit (not shown) that displays iron scrap classification information. Or when following other embodiments, those with ordinary knowledge in the relevant technical field can understand that some of the components illustrated in FIG. 1 may be omitted.

[0038] The iron scrap classification device 100 that provides iron scrap classification information according to a handling process can be used by a user and can be linked with all types of handheld-based wireless communication devices equipped with a touch screen panel, such as a mobile phone, smartphone, PDA (Personal Digital Assistant), PMP (Portable Multimedia Player), tablet PC, etc. In addition to this, it can also be included in or linked with devices provided with a base where an application can be installed and executed, such as a desktop PC, tablet PC, laptop PC, and IPTV including a set-top box.

[0039] The iron scrap classification device 100 that provides iron scrap classification information according to the handling process can be implemented on a terminal such as a computer that operates through a computer program for implementing the functions described in this specification.

[0040] The scrap iron classification device 100 that provides scrap iron classification information through a handling process according to an embodiment may include, but is not limited to, a system (not shown) and a related server (not shown) that provide classification information for scrap iron. The server according to an embodiment can assist an application that provides a service for providing classification information for scrap iron.

[0041] In the following, an embodiment will be mainly described in which the scrap iron classification device 100 that provides scrap iron classification information through a handling process according to an embodiment independently obtains and provides classification information results by a preset scrap iron classification method. However, as described above, it may also be carried out through cooperation with a server. That is, it can be understood that the scrap iron classification device 100 that provides scrap iron classification information through a handling process according to an embodiment and the server can be integrally implemented in terms of their functions, the server may be omitted, and it is not limited to any one embodiment.

[0042] In one embodiment, the scrap iron classification device 100 and the server can be linked, and the configuration of providing scrap iron classification information by performing a scrap iron classification process and a classification result information providing process can be carried out by the server or by the scrap iron classification device 100. For example, the scrap iron classification device 100 can operate as a server, and in the following, it will be uniformly described with the scrap iron classification device 100.

[0043] FIG. 2 is a flowchart showing each step in which the scrap iron classification device 100 according to an embodiment of the present disclosure provides scrap iron classification information through a handling process.

[0044] Referring to step S210, the iron scrap sorting device 100 according to an embodiment can acquire a loading state image captured during the handling process of a plurality of iron scraps by the loading equipment. In one embodiment, the iron scrap sorting device 100 can acquire a loading state image captured from the upper side to the lower side of the loading equipment. At this time, the loading equipment may be in a state where a plurality of iron scraps are loaded. Also, the loading state image is an image acquired during the handling process, and can include an image after at least one iron scrap is unloaded as time passes during the handling process and / or an image before the iron scrap is unloaded. Therefore, the iron scrap sorting device 100 can acquire a loading state image including a plurality of iron scraps during the handling process of at least one iron scrap.

[0045] Referring to step S220, the iron scrap sorting device 100 according to an embodiment can be updated as the handling process progresses and acquire layer information determined by the heights of a plurality of iron scraps. In one embodiment, the layer information can be information based on the heights of a plurality of iron scraps that change as time passes during the handling process. That is, the layer information can include position change information and / or height change information of a plurality of iron scraps updated by at least one iron scrap being unloaded from the loading equipment. In one embodiment, the method of acquiring the layer information can include a monotype using one camera or a stereo type using two or more cameras. In this regard, reference can be made to FIGS. 3 to 5 for description.

[0046] FIG. 3 is a drawing for explaining an example in which the iron scrap sorting device 100 according to an embodiment of the present disclosure acquires layer information based on the monotype.

[0047] Referring to FIG. 3, the monotype can utilize depth information obtained through analysis of changes in the wall surface of the loading platform obtained from an image acquired by one camera. For example, as the single camera of the iron scrap sorting device 100 moves, depth information indicating the heights of multiple iron scraps in the loading equipment can be obtained based on changes in the wall surface of the loading platform from the loading state images captured respectively in the vertical direction and / or one or more side surface directions.

[0048] FIG. 4 is a drawing for explaining an example in which the iron scrap sorting device 100 according to an embodiment of the present disclosure obtains layer information based on a stereotype.

[0049] Referring to FIG. 4, the stereotype can utilize depth information obtained through analysis of angular changes obtained from images acquired for the same region from two or more cameras located on the same plane. For example, the iron scrap sorting device 100 can obtain depth information indicating the heights of multiple iron scraps in the loading equipment due to image changes caused by angular changes for the same region in multiple loading state images captured in the vertical direction from two or more cameras located on the same plane. The iron scrap sorting device 100 can utilize an AI model to obtain layer information based on the depth information obtained based on the monotype and / or the stereotype. That is, the iron scrap sorting device 100 may perform analysis on changes in the wall surface of the loading platform using the monotype, or may perform analysis on image changes caused by angular changes using the stereotype. Therefore, the iron scrap sorting device 100 can obtain layer information based on depth information due to changes in the wall surface of the loading platform and depth information due to image changes in the region corresponding to the same region in the loading state image, and utilize it for image analysis, so the accuracy of the analysis can be improved. That is, limitations such as light reflection and shaking that may occur in the monotype using one camera can be complemented through the stereotype using two or more cameras.

[0050] FIG. 5 is a drawing for explaining an example in which the iron scrap classification apparatus 100 according to an embodiment of the present disclosure utilizes depth information.

[0051] Referring to FIG. 5, the loading state image shown at the upper end is an image acquired by a monotype, and the iron scrap classification apparatus 100 can acquire a loading state image captured in the vertical direction using an AI model. Further, a loading state image captured in the side direction can be acquired. Therefore, change analysis of the wall surface of the loading platform can be performed based on the loading state images captured in the vertical direction and the side direction. For example, a region with a larger change in the wall surface of the loading platform may be a region with a shallower depth, and a region with a smaller change in the wall surface to be loaded may be a region with a deeper depth. The iron scrap classification apparatus 100 can acquire a depth map showing the height or height change of a plurality of iron scraps shown at the lower end. As shown in FIG. 5, each of the plurality of iron scrap regions can be indicated by a different hue or a different hatched region according to the depth. That is, a region with a finer hatching density can indicate a deeper region. Therefore, the iron scrap classification apparatus 100 can acquire layer information including layer changes using the depth information of the depth map.

[0052] Referring to step S230, the iron scrap classification apparatus 100 according to an embodiment can determine a region of interest based on a loading state image that is updated as the loading process progresses. In one embodiment, the updated loading state image can be an image after at least one or more iron scraps have been unloaded as the loading process progresses, and the unloading region can correspond to the region of interest. In one embodiment, the iron scrap classification apparatus 100 can determine a region of interest based on the comparison result between the first loading state image corresponding to the first time point and the second loading state image corresponding to the second time point that is temporally subsequent to the first time point. For example, the first loading state image can correspond to an image before at least one or more iron scraps are unloaded, and the second loading state image can correspond to an image after at least one or more iron scraps are unloaded.

[0053] FIG. 6 is a drawing for explaining an example of determining a region of interest based on whether a grapple is included in any partial region of the vertical region of the loading equipment in the handling process of one or more iron scraps according to an embodiment of the present disclosure.

[0054] Referring to FIG. 6, in one embodiment, in the handling process of one or more iron scraps, a partial region including the grapple in the vertical region of the loading equipment can be determined as the region of interest. For example, a partial region corresponding to the case where the grapple appears and then disappears on the loading equipment (vertical region) can be determined as the region of interest.

[0055] FIG. 7 is a drawing for explaining an example of the iron scrap sorting device 100 according to an embodiment of the present disclosure obtaining a grapple operation region.

[0056] Referring to FIG. 7, the iron scrap sorting device 100 can obtain the grapple operation region of the image including the grapple from a plurality of loaded state images captured during the handling process. The iron scrap sorting device 100 can identify the grapple in the loaded state image using an AI model. As shown in FIG. 7, the grapple can be identified and the grapple operation region can be obtained. For example, the iron scrap sorting device 100 can obtain a certain region including the boundary of the grapple as the grapple operation region. Therefore, the iron scrap sorting device 100 can determine a partial region including the grapple operation region as the region of interest. The iron scrap sorting device 100 according to an embodiment can determine a partial region corresponding to the grapple operation region among the different regions between the first loaded state image and the second state image as the region of interest. The iron scrap sorting device 100 can compare and analyze the difference in the images between the region of interest corresponding to the grapple operation region in the first loaded state image and the region of interest corresponding to the grapple operation region in the second loaded state image.

[0057] Referring to stage S240, the iron scrap classification device 100 according to one embodiment can obtain a segmentation image for a target iron scrap that is included in the region of interest and is any one of a plurality of iron scraps. The iron scrap classification device 100 can use a segmentation model to obtain a segmentation image including the target iron scrap in the loaded state image. In one embodiment, the target iron scrap can be the iron scrap for which image analysis is to be performed. The iron scrap classification device 100 can perform segmentation on the iron scrap in any one of a plurality of iron scrap regions to obtain a segmentation image in order to classify each of the plurality of iron scraps included in the region of interest into individual iron scraps. In one embodiment, the segmentation image can include a plurality of images. For example, the iron scrap classification device 100 can perform instance segmentation on the loaded state image to obtain a segmentation image for a single object. Also, the iron scrap classification device 100 can perform semantic segmentation on the loaded state image to obtain a segmentation image for a clustered object. In one embodiment, the semantic segmentation can be performed on a pixel-by-pixel basis. For example, the iron scrap classification device 100 can obtain a segmentation region including at least one or more pixels corresponding to each of the plurality of iron scraps based on a pixel unit, and obtain a segmentation image that is an image corresponding to each segmentation region obtained in the region of interest. Therefore, the segmentation image can be a plurality of concepts including all of the segmentation image for a single object and the segmentation image for a clustered object. Semantic segmentation performs segmentation by recognizing it as a physically recognizable unit of meaning, and a segmentation image for a clustered object in which a plurality of objects are clustered can be obtained. Instance segmentation performs segmentation by recognizing each object as one unit, and a segmentation image for a single object can be obtained.

[0058] Referring to stage S250, the iron scrap classification device 100 according to one embodiment can obtain item information and grade information for the target iron scrap based on the divided image and layer information. The iron scrap classification device 100 can obtain item information and grade information corresponding to the target iron scrap by using a classification model that performs analysis in image units. In one embodiment, classification can be a process of analyzing or classifying iron scrap based on the feature information obtained from each image for each divided image. The iron scrap classification device 100 can perform analysis on each of the plurality of divided images in image units by using a classification model. Therefore, the iron scrap classification device 100 can obtain item information and grade information corresponding to each analysis image. In one embodiment, the item information corresponds to the characteristic information for the iron scrap and can include, for example, weight information, lightweight information, etc. Also, in one embodiment, the iron scrap classification device 100 can perform classification on the target iron scrap image to obtain grade information corresponding to the target iron scrap. That is, the iron scrap classification device 100 can obtain item information and grade information corresponding to each target iron scrap by performing classification. Also, the iron scrap classification device 100 can use layer information when performing classification. In one embodiment, the size of each of the plurality of iron scraps in the stacked state image can be shown differently by the layer. For example, the higher the stacking height for each of the plurality of iron scraps, the larger the size can be shown in the stacked state image. In one embodiment, since the size of the iron scrap can be an element that affects the item information or grade information, the iron scrap classification device 100 can use the depth information included in the layer information together when performing classification on the divided image to obtain item information and grade information. The depth information is an element that can confirm the height of each of the plurality of iron scraps and can be an element that indicates a more detailed height (depth) for determining the layer.The iron scrap classification device 100 can obtain item information and grade information for each of a plurality of iron scraps by using the classification results and the depth information of each of the plurality of iron scraps included in the divided image. Therefore, the iron scrap classification device 100 can not only use the image obtained planarly, but also further use the depth information to more precisely determine the item information and grade information in consideration of the size of the iron scrap according to the height (depth). As a result, the accuracy of the item information and grade information for the iron scrap can be increased.

[0059] Referring to step S260, the iron scrap classification device 100 according to an embodiment can provide iron scrap classification information including item information and grade information. As described above, the iron scrap classification device 100 can provide, as the iron scrap classification information, the item information and grade information for the target iron scrap obtained by using the segmentation model and the classification model.

[0060] FIG. 8 is a drawing for explaining an example in which the iron scrap classification device 100 according to an embodiment of the present disclosure obtains layer information by a monotype and obtains a unloading area based on a grapple operation area.

[0061] Referring to FIG. 8, the iron scrap sorting device 100 according to an embodiment can acquire the regions of interest or the regions of difference of the first loading state image and the second loading state image respectively as the unloading regions based on the grappling operation region described in FIG. 7. That is, the iron scrap sorting device 100 acquires the region of difference between the first loading state image showing the loading state image before at least one or more iron scraps are unloaded and the second loading state image showing the loading state image after at least one or more iron scraps are unloaded, measures the change for the region of difference, identifies the unloading region, and can determine a region of interest with higher accuracy. That is, the iron scrap sorting device 100 can identify the unloading region based on the grappling operation region and the region of difference. Therefore, the iron scrap sorting device 100 can determine a rectangular region (unloading region B-box) including the unloading region determined along the boundary indicating the region of difference as the final region of interest. Therefore, in order to acquire the divided image for the region of interest with respect to the unloading region determined along the boundary indicating the region of difference and acquire the iron scrap sorting information, there is an advantage that the error rate due to the change in the peripheral region other than the unloading region can be reduced. The iron scrap sorting device 100 according to an embodiment may determine the region of interest based on the operating state of the grapple as to whether the grapple contains at least one or more iron scraps when the grapple is included in a certain partial region of the vertical region of the loading equipment for a preset time or more. For example, the iron scrap sorting device 100 can determine the presence or absence of iron scrap unloading based on the time when the grapple is included in a certain partial region of the vertical region of the loading equipment. When the time when the grapple is included in a certain partial region is equal to or greater than the first time, the iron scrap sorting device 100 can determine that unloading has occurred for at least one or more iron scraps. In one embodiment, the first time may be a time preset to correspond to the case where unloading occurs for iron scraps during the loading and unloading process. Also, a certain partial region can include a plurality of regions that are updated as the grapple moves.Therefore, when the total time obtained by adding up the respective times during which a grapple is included in each of a plurality of partial regions of the iron scrap sorting device 100 is equal to or longer than the first time, it can be determined that unloading of the iron scrap has occurred. Further, the iron scrap sorting device 100 can determine, as an unloading region, a partial region among the plurality of partial regions in which the time during which the grapple is included is the longest. That is, in one embodiment, the first time can be a reference time for determining the presence or absence of the occurrence of unloading of the iron scrap. In the iron scrap sorting device 100 according to one embodiment, when the grapple is included in a partial region for a second time that is equal to or longer than a preset time (for example, half of the first time) longer than the first time, it is possible to obtain the operating state of the grapple with respect to whether each of a plurality of loading state images captured during the handling process includes at least one iron scrap in the grapple. When the time during which the grapple is included in a partial region is equal to or longer than the second time, it may be a situation where it can be determined that unloading has not occurred. For example, the second time is a time that is equal to or longer than the preset time longer than the first time, and when the grapple is included in a partial region and the included time is equal to or longer than the second time, there is a high possibility of corresponding to a situation where the current operation of the grapple is being performed in real time. Therefore, the iron scrap sorting device 100 can obtain the operating state of the grapple and can determine a region of interest based on the operating state. For example, by obtaining the operating state of the grapple, it is possible to identify a loading state image in which the grapple includes at least one iron scrap among a plurality of loading state images. The iron scrap sorting device 100 can determine, as a region of interest, a certain boundary region where the grapple is located at the time when the operating state of the grapple first includes an iron scrap, or can determine, as a region of interest, a certain boundary region where the grapple is located at the time when the operating state of the grapple last includes an iron scrap. When the time during which the grapple is included in a partial region is equal to or longer than the second time, it can correspond to a case where the grapple stays in the vertical direction region of the loading equipment for a long time.That is, when there is an image of the loaded state in which the grapple contains at least one or more iron scraps, it can be adapted to the case of changing the positions of a plurality of iron scraps rather than unloading a plurality of iron scraps with the loading equipment. Therefore, the iron scrap classification device 100 may determine the region of interest based on the first and last points in time when the grapple contains iron scraps. In the case of the region of interest corresponding to the first point in time, it can be analyzed to be similar to the state where some of the iron scraps have been unloaded, and in the case of the region of interest corresponding to the last point in time, it can be analyzed to be similar to the state where some of the iron scraps have been loaded. Therefore, the iron scrap classification device 100 can determine the region of interest for the region where the iron scraps are unloaded based on the case where the grapple stays in the vertical region of the loading equipment and then disappears, and when the grapple stays in the vertical region of the loading equipment for more than a second time, it can determine the region of interest for the region where the iron scraps are moved based on the grapple operating state. Therefore, there is an effect that analysis and classification for each of a plurality of iron scraps that can be positioned according to various situations can be performed. In another embodiment, the iron scrap classification device 100 can obtain an operation pattern based on the grapple operating state. For example, when the time during which the grapple is included in a part of the region is more than a second time, the iron scrap classification device 100 analyzes a plurality of loaded state images that are continuously captured to determine whether there is a repetition of the grapple operating state including iron scraps and the grapple operating state not including iron scraps. When an operation pattern in which the state including iron scraps and the state not including iron scraps repeatedly occur is obtained, the iron scrap classification device 100 may differently determine the position and number of the regions of interest based on the sequential positions of the state including iron scraps and the state not including iron scraps as the number of repetitions and the passage of time.For example, when the number of repetitions of the iron scrap classification device 100 is 1, since the positions of some of the plurality of iron scraps are moved once, the number of regions of interest is determined to be 2, and the positions of the regions of interest are set as the region corresponding to the position of the grapple at the time when the grapple first includes the iron scrap and the region corresponding to the position of the grapple at the time when the grapple last includes the iron scrap. Also, the size of the region of interest can be determined by the change between the previous loaded state image (the image before being updated) and the current loaded state image (the updated image). When the number of repetitions of the iron scrap classification device 100 is 1, the region of interest may be updated based on the overlapping probability for the two regions of interest. For example, when the sequential positions of the state including the iron scrap and the state not including the iron scrap are less than a preset distance (for example, less than the length corresponding to 10% of the lateral length of the loading equipment), the two regions of interest are overlapped and updated to one region of interest to perform a primary image analysis, a secondary image analysis is performed on the two regions of interest, the two regions of interest are subdivided and updated to three regions of interest to perform a tertiary image analysis. That is, when the sequential positions of the state including the iron scrap and the state not including the iron scrap are less than the preset distance, the iron scrap classification information can be provided through the tertiary image analysis process. When corresponding to less than the preset distance, since the overlapping probability for the region of interest is high, the two regions of interest can be overlapped and determined as one region of interest, and the item information and grade information for each target iron scrap, which is any one of the plurality of iron scraps, can be obtained based on the layer information. Also, analysis can be performed on each of the two regions of interest to obtain the item information and grade information for each target iron scrap. Also, analysis can be performed on each of the three regions of interest divided based on the distance between the midpoints of the grapples in the state including the iron scrap and the state not including the iron scrap and the length that is 1 / 3 of the average of the preset distance to obtain the item information and grade information for each target iron scrap.Therefore, the analysis accuracy for each target iron scrap can also be improved by comparing and analyzing the results of the three-dimensional image analysis process. In other embodiments, when the number of repetitions of the iron scrap classification device 100 is two or more, as described above, it can be determined that it is not a situation where the process of analyzing so that some iron scraps are in a state similar to being unloaded in one region of interest and analyzing so that some iron scraps are in a state similar to being loaded in another region of interest is applied. That is, when the number of repetitions is two or more, it can be determined that it corresponds to the act of mixing a plurality of iron scraps and does not correspond to unloading and / or loading. Therefore, when the number of repetitions is two or more, the iron scrap classification process may be terminated, and when the grapple is not included in some regions (when the grapple disappears), it may be reset and the iron scrap classification process may be re-executed.

[0062] FIG. 9 is a drawing for explaining an example in which the iron scrap classification device 100 according to an embodiment of the present disclosure obtains layer information at different layer change times by a stereotype.

[0063] Referring to FIG. 9, the iron scrap classification device 100 according to an embodiment can obtain the time point of the first layer change at which the height of a plurality of iron scraps changes to be equal to or greater than the critical length based on the depth information acquired by stereotyping. Further, the iron scrap classification device 100 can obtain the time point of the second layer change at which the volume of a plurality of iron scraps changes to be equal to or greater than the critical percentage based on the average volume of the loading equipment. Therefore, the iron scrap classification device 100 can obtain layer information based on at least one of the time point of the first layer change and the time point of the second layer change. In one embodiment, the iron scrap classification device 100 can perform a layer change measurement process. A layer can be a unit concept corresponding to a layer of iron scraps that has changed after at least one or more iron scraps have been unloaded using a grapple. For example, when the loading equipment is loaded with iron scraps to the brim, it can be referred to as Layer-3, and when iron scraps are loaded exceeding the height of the loading equipment, it can be referred to as Layer-3.5. Also, during the handling process, when the height of the iron scraps on the loading equipment changes to be equal to or greater than the critical length, the layer can be updated to a smaller layer (for example, Layer-2 or Layer-1) one by one. In one embodiment, when determining item information and grade information for the target iron scrap using the layer information determined based on the depth information acquired by stereotyping, it is insensitive to changes due to movement, etc., and since there is time between layers, there is an advantage that a higher level of AI or machine learning can be applied. As shown in FIG. 9, the iron scrap classification device 100 can determine the layer as Layer-3 when a plurality of iron scraps are full in the loading equipment, and can determine a lower layer as the loaded height decreases. It can be confirmed that in Layer-1 of FIG. 9, the height at which a plurality of iron scraps are loaded is low and the volume of the plurality of iron scraps is small, and in Layer-3 of FIG. 9, it can be confirmed that the height at which a plurality of iron scraps are loaded is high and the volume of the plurality of iron scraps is large. The iron scrap classification device 100 can obtain a depth map for each layer to obtain depth information.In one embodiment, when the average unloading time is around 5 minutes (300 seconds), the unloading progresses about 20 times with a grapple, and the layer is approximately 3 - layer, when determining item information and grade information in grapple units, it must be inferred within 15 seconds (300 seconds / 20 times). On the contrary, when determining item information and grade information in layer units, it can be determined within 100 seconds (300 seconds / 3 - layer). Therefore, there is an advantage that a higher level of AI or machine learning can be applied. In one embodiment, the iron scrap classification device 100 can determine the time point when the height of a plurality of iron scraps becomes lower than the critical length (about 0.4m) as the first layer change time point. Also, the iron scrap classification device 100 can obtain average volume information for the loading equipment from the administrator account. The iron scrap classification device 100 can determine the time point when the volume of a plurality of iron scraps decreases by more than the critical percentage (about 33%) based on the average volume of the loading equipment as a reference based on the obtained average volume information as the second layer change time point. Therefore, the iron scrap classification device 100 can determine the layer at either the earliest time point in chronological order based on the first layer change time point and / or the second layer change time point or the average time point of the first layer change time point and the second layer change time point. When analyzing the divided image, the iron scrap classification device 100 can further utilize the depth information obtained in at least one or more determined layers to obtain item information and grade information for the target iron scrap. That is, the change between each image updated by layer based on the layer change time point can be measured. In another embodiment, the iron scrap classification device 100 can assign different weight values to the first layer change time point and the second layer change time point. For example, since the importance of the length of the height (depth) may be higher than the importance of the volume when using layer information, weight values can be assigned to the first layer change time point and the second layer change time point at a ratio of 7:3 respectively. Also, the iron scrap classification device 100 may determine the ratio differently depending on the initial loading state in which a plurality of iron scraps are loaded.For example, the importance of height (depth) can be determined differently according to the height loaded in the initial loading state. For example, when a plurality of iron scraps are loaded so as to correspond to the full height of the loading equipment based on the initial loading state, weight values can be assigned at a ratio of 8:2 at the time of the first layer change and the time of the second layer change. When a plurality of iron scraps are loaded so as to correspond to 90% or more of the height of the loading equipment, weight values can be assigned at a ratio of 7:3 at the time of the first layer change and the time of the second layer change. When a plurality of iron scraps are loaded so as to correspond to 80% or more and less than 90% of the height of the loading equipment, weight values can be assigned at a ratio of 6:4 at the time of the first layer change and the time of the second layer change. When a plurality of iron scraps are loaded so as to correspond to less than 80% of the height of the loading equipment, weight values can be assigned at a ratio of 8:2 at the time of the first layer change and the time of the second layer change and can be assigned at a ratio of 5:5. Therefore, since the ratio at which weight values are assigned is updated to increase the importance of the length-to-volume ratio according to the loading height of the loading equipment and the average time point at which the layer is acquired is determined, the efficiency can be improved when using the layer information. In another embodiment, the iron scrap classification device 100 may determine the importance of volume differently according to the size of the loading equipment. For example, when the overall volume of the loading equipment is small and less than the preset volume, the sensitivity of the volume may be high, so weight values may be assigned at a ratio of 3:7 at the time of the first layer change and the time of the second layer change. Also, when the overall volume of the loading equipment is large and is a certain multiple (3 times) or more of the preset volume, the critical length and the critical percentage may be updated. For example, when the overall volume of the loading equipment is large and is a certain multiple or more, even if a large amount of iron scraps are unloaded, the height may not decrease much. Therefore, the critical length can be reduced to a length corresponding to 80% of the conventional critical length (for example, about 0.32 m), and the critical percentage can be increased to a percentage corresponding to 120% of the conventional critical percentage (for example, about 39.6%). Therefore, since the sensitivity of height and volume is adjusted according to the situation, the efficiency can be improved. Each of the numerical values and percentages described above can be changed flexibly according to the situation and may be changed and set by the administrator.

[0064] FIG. 10 is a drawing schematically showing an example in which an iron scrap classification apparatus 100 according to an embodiment of the present disclosure performs semantic segmentation and instance segmentation.

[0065] Referring to FIG. 10, the iron scrap classification apparatus 100 according to an embodiment can perform semantic segmentation that divides the same type of objects in the same region and / or instance segmentation that divides the same type of objects into different regions when dividing the types of objects in the image and their boundary lines. That is, the AI learning model can obtain the types and positions of iron scraps in the loaded state image, and can obtain the item information and grade information corresponding to each iron scrap. The iron scrap classification apparatus 100 according to an embodiment can perform semantic segmentation and instance segmentation in parallel. That is, instance segmentation may be performed after semantic segmentation is performed, or semantic segmentation may be performed after instance segmentation is performed. For example, as shown in FIG. 10, the iron scrap classification apparatus 100 can perform semantic segmentation that divides the same type of objects in the same region, and instance segmentation that divides each of the divided images of the clustered objects after semantic segmentation into different iron scrap regions one by one to obtain a divided image of a single object. As shown in FIG. 10, the iron scrap classification apparatus 100 can perform semantic segmentation that labels (divides) a plurality of iron scraps into clustered objects, and can perform instance segmentation that labels (divides) a plurality of iron scraps into respective single objects. Conversely, after instance segmentation is performed and image segmentation for a single object is performed, the iron scrap classification apparatus 100 may perform semantic segmentation on the region excluding the region corresponding to the single object. Therefore, there is an effect that the accuracy can be further improved because a divided image of the clustered objects can be further obtained even for a region where instance segmentation is not accurately performed. This can be described with reference to FIG. 11.

[0066] FIG. 11 is a drawing schematically showing an example in which the iron scrap classification apparatus 100 according to an embodiment of the present disclosure further acquires a divided image for a clustered object with respect to a region from which a region corresponding to a single object has been excluded.

[0067] Referring to FIG. 11, the iron scrap classification apparatus 100 according to an embodiment can perform semantic segmentation on a region from which a region corresponding to a single object acquired by instance segmentation in FIG. 10 has been excluded, to obtain a divided image for a clustered object. Therefore, instance segmentation may be further performed on the region corresponding to the clustered object after performing semantic segmentation. Thus, as shown at the lower end of FIG. 11, the regions of each of a plurality of iron scraps can be further divided into single objects even with respect to the region from which the region corresponding to the single object has been excluded. Although an example in which semantic segmentation and instance segmentation are performed on a plurality of iron scraps in the entire loading equipment in FIG. 11 has been described, the present disclosure is not limited thereto, and semantic segmentation and instance segmentation can be equally applied to a region of interest. In this way, the iron scrap classification apparatus 100 obtains a divided image for each region of a plurality of iron scraps by using semantic segmentation and instance segmentation in parallel, so that there is an effect that the accuracy of item information and grade information for each target iron scrap can be improved.

[0068] According to one embodiment, there is an effect that the accuracy of iron scrap classification can be improved by obtaining segmented images for single objects and cluster objects respectively in terms of performing image analysis and image classification using a segmentation model and a classification model. Also, in terms of obtaining layer information based on monotypes and stereotypes, the accuracy of image classification and analysis for the unloading area can be increased, and in terms of determining an area of interest based on the grapple operation area, there is an effect that the accuracy of detecting the unloading area can be increased. Further, in terms of obtaining a segmented image by using semantic segmentation and instance segmentation in parallel, there is an advantage that it is possible to complement areas where each segmentation is not accurately performed.

[0069] Various embodiments of the present disclosure may be embodied in software including one or more instructions stored in a machine (e.g., a display device or a computer)-readable storage medium (e.g., a memory). For example, a processor (e.g., processor 220) of the machine may call and execute at least one of the one or more instructions stored from the storage medium. This enables the machine to be operated to perform at least one function by the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" only means that the storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), and this term does not distinguish between cases where data is stored semi-permanently and temporarily in the storage medium.

[0070] According to an embodiment, the methods according to the various embodiments disclosed in the present disclosure can be provided included in a computer program product. The computer program product can be traded as a commodity between a seller and a purchaser. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or through an application store (e.g., Play Store™), or directly online (e.g., download or upload) between two user devices (e.g., smartphones). In the case of online distribution, at least a part of the computer program product can be at least temporarily stored or temporarily generated in a machine-readable storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0071] Although the present invention has been described with reference to the drawings illustrated, it is not limited by the disclosed embodiments and the drawings. It will be understood that those having ordinary knowledge in the technical field related to this embodiment can be embodied in a modified form without departing from the essential characteristics of the above description. Therefore, the disclosed method should be considered from an explanatory perspective rather than a limiting perspective. Even if the effects related to the configuration of the present invention are not explicitly described when explaining the embodiments, the effects that can be predicted by the corresponding configuration can also be recognized. The scope of the present invention is shown not in the above description but in the claims, and all differences within the equivalent scope should be construed as being included in the present invention.

Explanation of Reference Numerals

[0072] 100: Iron Scrap Classification Device 110: Receiver 120: Processor

Claims

1. In a method for providing iron scrap classification information based on a handling process, a step of a receiving unit obtaining a loading state image captured during a handling process for a plurality of iron scraps loaded on a handling equipment; a step of a processor obtaining layer information updated as the handling process progresses and determined by the height of the plurality of iron scraps; a step of the processor determining a region of interest based on the loading state image updated as the handling process progresses; a step of the processor obtaining a segmented image for a target iron scrap that is one of the plurality of iron scraps and is included in the region of interest; a step of the processor obtaining item information and grade information for the target iron scrap based on the segmented image and the layer information; and a step of the processor providing iron scrap classification information including the item information and the grade information; A method comprising.

2. The method of obtaining the layer information includes a monotype using one camera or a stereo type using two or more cameras, The method according to claim 1, wherein the monotype utilizes depth information obtained through analysis of changes in the wall surface of the loading platform obtained from an image obtained by the one camera.

3. The step of determining the region of interest is The method according to claim 1, wherein the processor determines the region of interest based on a comparison result between a first loading state image corresponding to a first time point and a second loading state image corresponding to a second time point that is temporally subsequent to the first time point.

4. The step of determining the region of interest is The method according to claim 3, wherein the processor determines the region of interest based on a difference region between the first loading state image and the second loading state image and an operation region of a grapple used in the handling process.

5. The step of the processor obtaining the segmented image is to obtain the segmented image including the target iron scrap from the loading state image using a segmentation model, The method according to claim 1, wherein the step of the processor obtaining the item information and the grade information is to obtain the item information and the grade information corresponding to the target iron scrap using a classification model that performs analysis on an image-by-image basis.

6. The step of obtaining the segmented image is The step of the processor performing instance segmentation on the loaded state image to obtain the segmented image for a single object; and The step of the processor performing semantic segmentation on the loaded state image to obtain the segmented image for a clustered object; The method according to claim 5, comprising:

7. The step of obtaining the segmented image for the clustered object is The method according to claim 6, which is performed on a region of the loaded state image from which a region corresponding to the single object has been excluded.

8. The method according to claim 2, wherein the stereotype utilizes depth information obtained through analysis of angle changes obtained from images acquired for the same region from two or more cameras located on the same plane.

9. The second time point is determined based on whether the grapple is included in any partial region of the vertical region of the loading equipment after the first time point, The step of determining the region of interest is The method according to claim 4, wherein when the grapple is included in the partial region for a preset time or more, the processor determines the region of interest based on the operating state of the grapple as to whether the grapple includes at least one or more iron scraps.

10. The method of obtaining the layer information is The processor obtains a first layer change time point at which the height of the plurality of iron scraps changes to a critical length or more based on the depth information obtained by the stereotype, The processor obtains a second layer change time point at which the volume of the plurality of iron scraps changes to a critical percentage or more based on the average volume of the loading equipment, The method according to claim 8, wherein the processor obtains the layer information based on at least one of the first layer change time point and the second layer change time point.

11. In an iron scrap classification device that provides iron scrap classification information based on a handling process, A receiving unit that obtains a loaded state image captured during a handling process for a plurality of iron scraps loaded on a loading equipment; and Layer information that is updated as the handling process progresses and is determined by the height of the plurality of iron scraps is obtained, A region of interest is determined based on the loaded state image that is updated as the handling process progresses, Obtain a segmented image of a target iron scrap that is included in the region of interest and is any one of the plurality of iron scraps. Obtain item information and grade information for the target iron scrap based on the segmented image and the layer information. A processor that provides iron scrap classification information including the item information and the grade information; An iron scrap classification device including the same.

12. The method for obtaining the layer information Includes a monotype using one camera or a stereo type using two or more cameras. The iron scrap classification device according to claim 11, wherein the monotype uses depth information obtained through analysis of changes in the wall surface of the loading platform obtained from an image obtained by the one camera.

13. The processor Determines the region of interest based on a comparison result between a first loading state image corresponding to a first time point and a second loading state image corresponding to a second time point that is temporally subsequent to the first time point. The iron scrap classification device according to claim 11.

14. The processor Determines the region of interest based on a difference region between the first loading state image and the second loading state image and an operation region of a grapple used in the handling process. The iron scrap classification device according to claim 13.

15. A computer-readable recording medium recording a program for causing a computer to execute the method according to any one of claims 1 to 10.

Images