Method for providing iron scrap classification information, iron scrap classification apparatus, and recording medium

Abstract

The invention relates to a method for providing iron scrap classification information, an iron scrap classification apparatus, and a recording medium. The method includes: obtaining, by a receiving unit, a loading status image captured during an unloading process of a plurality of iron scraps loaded onto a loading device; obtaining, by the processor, layer information updated as the unloading process proceeds and determined from the height of the plurality of iron scraps; determining, by the processor, a region of interest based on the load state image updated as the unload process proceeds; obtaining, by the processor, a segmented image of a target iron scrap included in the region of interest and being any one of the plurality of iron scraps; obtaining, by a processor, article information and grade information about the target iron scrap based on the segmented image and the layer information; and providing, by the processor, classification information about the iron scrap including the item information and the grade information.

Description

Technical Field

[0001] The technical field of the present disclosure relates to a method of providing result information on image classification according to a process of unloading an object, a single object, etc. loaded on a loading device, and to a technical field of a method of providing classification information on iron scrap according to image analysis of an area where one or more iron scraps are unloaded. Background Art

[0002] Recently, with the growth of the logistics industry, loading devices are widely used to load various objects or move objects from one place to another. However, multiple objects loaded on a loading device can be managed by a worker's decision or loaded at positions for each product or specification, but it may be difficult to check the positions every time. Therefore, a segmentation method can be used in which the entire area of a loading device loaded with multiple objects is divided to monitor images of each area. Generally, in a segmentation method for image analysis related to iron scrap, an optical system is easily installed at the unloading position to obtain an image without hardware (H / W) engineering that sets the position and angle of the optical system (closed-circuit television (CCTV) and mechanism) so that artificial intelligence (AI) can perform optimally. In this case, there are limitations in that a large amount of image collection and labeling are required because methods for classifying irregular iron scraps having few consistent features (various shapes, ordinary textures, various colors (painted, rusted, etc.) depending on the usage method) based on instance segmentation and various cutting methods, bending, etc. are adapted as methods for determining the grade of iron scrap. Therefore, there is a need to provide a service in which performance can be improved by using only a relatively small amount of image collection and labeling by providing an image analysis method and system that can solve the limitations of an analysis method that requires such a large amount of image collection.

[0003] [Prior Art Documents]

[0004] [Patent Documents]

[0005] Korean Unexamined Patent Application Publication No. 10-2011-0078566 (published on July 7, 2011): Efficient Object Loading Location Detection System Using Digital Image Recognition. Summary of the Invention

[0006] The present disclosure relates to providing a technology for improving the accuracy of image analysis and classification of one or more iron scraps included in a loading state image when providing classification information on iron scraps according to an unloading process, and providing a service for obtaining a segmented image of an area determined to be an unloading area from a loading state image captured during the unloading process and providing high-precision image classification information on the segmented image.

[0007] The object of the present disclosure is not limited to the above object, and there may be other technical objects.

[0008] According to an aspect of the present disclosure, there is provided a method for providing classification information on iron scraps according to an unloading process, the method including: obtaining, by a receiving unit, a loading state image captured during the unloading process of a plurality of iron scraps loaded onto a loading device; obtaining, by a processor, layer information that is updated as the unloading process progresses and is determined according to the height of the plurality of iron scraps; determining, by the processor, a region of interest based on the loading state image updated as the unloading process progresses; obtaining, by the processor, a segmented image of a target iron scrap included in the region of interest and being any one of the plurality of iron scraps; obtaining, by the processor, article information and grade information on the target iron scrap based on the segmented image and the layer information; and providing, by the processor, classification information on the iron scraps including the article information and the grade information.

[0009] The method for obtaining the layer information may include a single-type method using one camera or a stereo-type method using two or more cameras, and in the single-type method, depth information obtained by analyzing changes in the wall surface of the loading box obtained from an image obtained by one camera may be used.

[0010] The step of determining the region of interest may include: determining, by the processor, the region of interest based on a result of comparing a first loading state image corresponding to a first time point with a second loading state image corresponding to a second time point later in time than the first time point.

[0011] The step of determining the region of interest may include: determining, by the processor, the region of interest based on a difference region between the first loading state image and the second loading state image and an operation region of a grapple used in the unloading process.

[0012] The step of obtaining, by the processor, the segmented image may include: obtaining, using a segmentation model, a segmented image including the target iron scrap from the loading state image, and the step of obtaining, by the processor, the article information and the grade information may include obtaining the article information and the grade information corresponding to the target iron scrap using a classification model that performs analysis on an image-by-image basis.

[0013] The steps of obtaining the segmented image may include the processor performing instance segmentation on the loading state image to obtain the segmented image of a single object, and the processor performing semantic segmentation on the loading state image to obtain the segmented image of the aggregated object.

[0014] Obtaining the segmented image of the aggregated object may be performed on a region of the loading state image excluding the region corresponding to the single object.

[0015] In the stereoscopic method, depth information obtained by analyzing the angular change of images obtained for the same region may be used, and the images are obtained from two or more cameras located on the same plane.

[0016] The second time point may be determined based on whether the grapple is included in any partial region of the region in the vertical direction of the loading device after the first time point, and the step of determining the region of interest may include: when the grapple is included in the partial region for a preset time period or longer, the processor determining the region of interest based on the operating state of the grapple indicating whether the grapple includes at least one iron scrap.

[0017] The steps of obtaining the layer information may include: the processor obtaining a first layer change time point at which the height of a plurality of iron scraps is changed to a critical height or higher based on the depth information obtained according to the stereoscopic method; the processor obtaining a second layer change time point at which the volume of a plurality of iron scraps is changed to a critical percentage or higher based on the average volume of the loading device; and the processor obtaining the layer information at at least one of the first layer change time point and the second layer change time point.

[0018] According to another aspect of the present disclosure, there is provided an iron scrap classification device for providing classification information about iron scraps according to an unloading process. The iron scrap classification device includes: a receiving unit configured to obtain a loading state image captured during the unloading process of a plurality of iron scraps loaded on a loading device; and a processor configured to: obtain layer information that is updated as the unloading process progresses and is determined according to the height of the plurality of iron scraps; determine a region of interest based on the loading state image updated as the unloading process progresses; obtain a segmented image of a target iron scrap included in the region of interest and being any one of the plurality of iron scraps; obtain item information and grade information about the target iron scrap based on the segmented image and the layer information; and provide classification information about the iron scrap including the item information and the grade information.

[0019] The method of obtaining the layer information may include a single-type method using one camera or a stereo-type method using two or more cameras, and in the single-type method, depth information obtained by analyzing changes in the wall surface of the loading box obtained from the image obtained by the one camera may be used.

[0020] The processor may determine the region of interest based on a result of comparing a first loading state image corresponding to a first time point with a second loading state image corresponding to a second time point later in time than the first time point.

[0021] The processor may determine the region of interest based on a difference region between the first loading state image and the second loading state image and an operation region of a grapple used during the unloading process.

[0022] According to another aspect of the present disclosure, there is provided a computer-readable non-transitory recording medium on which a program for implementing the method of the first aspect is recorded. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] By referring to the accompanying drawings and describing in detail exemplary embodiments of the present disclosure, the above and other objects, features, and advantages of the present disclosure will become more apparent to those of ordinary skill in the art. In the drawings:

[0024] Figure 1 is a block diagram schematically showing a configuration of an iron scrap sorting device for providing classification information about iron scrap according to an embodiment of the present disclosure;

[0025] Figure 2 is a flowchart showing an operation of an iron scrap sorting device providing classification information about iron scrap according to an unloading process according to an embodiment of the present disclosure;

[0026] Figure 3 is a diagram for describing an example in which an iron scrap sorting device obtains layer information based on a single-type method according to an embodiment of the present disclosure;

[0027] Figure 4 is a diagram for describing an example in which an iron scrap sorting device obtains layer information based on a stereo-type method according to an embodiment of the present disclosure;

[0028] Figure 5 is a diagram for describing an example in which an iron scrap sorting device uses depth information according to an embodiment of the present disclosure;

[0029] Figure 6FIG. 0 is a diagram for describing an example in which an iron scrap sorting device according to an embodiment of the present disclosure determines a region of interest based on whether a grapple is included in any partial region of a vertical direction region of a loading device;

[0030] Figure 7 FIG. 4 is a diagram for describing an example in which an iron scrap sorting device according to an embodiment of the present disclosure obtains a grapple operation region;

[0031] Figure 8 FIG. 8 is a diagram for describing an example in which an iron scrap sorting device according to an embodiment of the present disclosure obtains layer information according to a single-type method and obtains an unloading region based on a grapple operation region;

[0032] Figure 9 FIG. 12 is a diagram for describing an example in which an iron scrap sorting device according to an embodiment of the present disclosure obtains layer information at each layer change time point according to a three-dimensional method;

[0033] Figure 10 FIG. 16 is a diagram for describing an example in which an iron scrap sorting device according to an embodiment of the present disclosure performs semantic segmentation and instance segmentation; and

[0034] Figure 11 FIG. 20 is a diagram for describing an example in which an iron scrap sorting device according to an embodiment of the present disclosure also obtains a segmented image of an aggregated object excluding a region corresponding to a single object; DETAILED DESCRIPTION

[0035] Advantages and features of the present disclosure and methods for implementing the same will be clearly understood with reference to the accompanying drawings and embodiments described in detail below. However, the present disclosure is not limited to the embodiments to be disclosed below, but may be implemented in various different forms. The embodiments are provided to fully explain the present embodiments and fully explain the scope of the present disclosure to those skilled in the art.

[0036] The terms used herein are provided only for describing embodiments of the present disclosure and are not for the purpose of limitation. In this specification, unless the context clearly indicates otherwise, the singular form includes the plural form. It should be understood that the terms "comprising" and / or "including" as used herein specify the presence of some of the described components, but do not exclude the presence or addition of one or more other components. Throughout the specification, the same reference numerals denote the same components, and "and / or" includes each combination of one or more of the above components. It should be understood that although the terms "first", "second", etc. may be used herein to describe various components, these components are not limited by these terms. These terms are only used to distinguish one component from another. Therefore, it should be understood that the first component described below may be the second component within the technical scope of the present disclosure.

[0037] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. Additionally, it should be further understood that terms such as those defined in common dictionaries should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0038] Spatial relative terms such as "below", "beneath", "lower", "above", "upper", etc. may be used to facilitate description of the relationship between one component and other components as shown in the drawings. The spatial relative terms should be understood to include different orientations of the element during use or operation in addition to the orientation shown in the drawings. For example, when the component shown in the drawings is flipped, a component described as "below" or "beneath" another component may ultimately be placed "above" the other component. Thus, the exemplary term "below" can include both a downward and an upward direction. The components may be arranged in different orientations such that the spatial relative terms can be interpreted according to the arrangement.

[0039] Hereinafter, embodiments will be described in detail with reference to the drawings.

[0040] Figure 1 is a block diagram schematically showing the configuration of an iron scrap sorting device 100 for providing classification information about iron scrap according to an embodiment of the present disclosure.

[0041] Referring to Figure 1 , the iron scrap sorting device 100 may include a receiving unit 110 and a processor 120.

[0042] According to one embodiment, the receiving unit 110 may obtain a loading state image captured during the unloading process of a plurality of iron scraps loaded onto a loading device.

[0043] The processor 120 according to one embodiment may obtain layer information that is updated as the unloading process progresses and determined based on the height of the plurality of iron scraps. Additionally, the processor 120 may determine a region of interest based on the loading state image updated as the unloading process progresses. Additionally, the processor 120 may obtain a segmented image of a target iron scrap that is included in the region of interest and is any one of the plurality of iron scraps. Additionally, the processor 120 may obtain item information and grade information about the target iron scrap based on the segmented image and the layer information. Additionally, the processor 120 may provide classification information about the iron scrap including the item information and the grade information.

[0044] In addition, the iron scrap classification device 100 for providing classification information on iron scrap according to the unloading process may be combined with various conventional networks such as the Internet, mobile communication networks, etc. during the process in which the receiving unit 110 obtains a loading state image and the processor 120 obtains layer information based on the heights of a plurality of iron scraps, determines a region of interest based on the loading state image, obtains a segmented image of the target iron scrap, and provides classification information on the iron scrap for the segmented image. It should be noted that there is no special limitation on the network.

[0045] In addition, those skilled in the art should understand that other general components in addition to Figure 1 the components shown may also be included in the iron scrap classification device 100 for providing classification information on iron scrap according to the unloading process through image analysis. For example, the iron scrap classification device 100 for providing classification information on iron scrap according to the unloading process may further include a memory (not shown) for storing the loading state image, layer information, region of interest, segmented image, article information and grade information corresponding to the segmented image, etc., and may also include a sending unit (not shown) for providing classification information on the iron scrap or a display unit (not shown) for displaying classification information on the iron scrap. Alternatively, those skilled in the art will understand that in another embodiment, some components shown in Figure 1 may be omitted.

[0046] The iron scrap classification device 100 for providing classification information on iron scrap according to an embodiment can be used by a user, can be linked with any type of handheld device-based wireless communication device equipped with a touch screen panel (such as a mobile phone, smart phone, personal digital assistant (PDA), portable multimedia player (PMP), tablet computer, etc.), and in addition, can be included in or linked with a device having a basis for installing and executing applications (such as a desktop personal computer (PC), tablet computer, laptop computer, Internet protocol television (IPTV) including a set-top box).

[0047] The iron scrap classification device 100 for providing classification information on iron scrap according to the unloading process can be implemented as a terminal such as a computer that operates through a computer program to implement the functions described in this specification.

[0048] The iron scrap classification device 100 for providing classification information on iron scrap according to an embodiment may include a system (not shown) for providing classification information on iron scrap and a related server (not shown), but the present disclosure is not limited thereto. A server according to an embodiment may support an application for providing a service of providing classification information on iron scrap.

[0049] Hereinafter, the iron scrap sorting device 100 for providing sorting information on iron scrap according to an unloading process independently obtains and provides the result of sorting information according to a preset method for sorting iron scrap. However, as described above, the iron scrap sorting device 100 for providing sorting information on iron scrap according to an unloading process may perform the above functions in combination with a server. That is, the iron scrap sorting device 100 and the server according to an embodiment may be implemented in an integrated manner in terms of their functions, and the server may be omitted, and it can be seen that the present disclosure is not limited to any one embodiment.

[0050] In one embodiment, the iron scrap sorting device 100 and the server may be linked to each other, and by performing an iron scrap sorting process and a sorting result information providing process, the server or the iron scrap sorting device 100 may perform a configuration for providing sorting information on iron scrap. For example, the iron scrap sorting device 100 may operate as a server, and the iron scrap sorting device 100 and the server are hereinafter collectively referred to as the iron scrap sorting device 100.

[0051] Figure 2 It is a flowchart showing the corresponding operations of the iron scrap sorting device 100 according to an embodiment of the present disclosure for providing sorting information on iron scrap according to an unloading process.

[0052] Referring to operation S210, the iron scrap sorting device 100 according to an embodiment may obtain a loading state image captured during the unloading process of a plurality of iron scraps loaded on a loading device. In one embodiment, the iron scrap sorting device 100 may obtain a loading state image captured from the upper side of the loading device, and in this case, a plurality of iron scraps are loaded on the loading device. In addition, the loading state image is an image obtained during the unloading process, and may include an image after unloading one or more iron scraps and / or an image before unloading one or more iron scraps over time during the unloading process. Therefore, the iron scrap sorting device 100 may obtain a loading state image including a plurality of iron scraps during the unloading process of one or more iron scraps.

[0053] Referring to operation S220, the iron scrap sorting device 100 according to one embodiment may obtain layer information, which is updated as the unloading process progresses and is determined based on the heights of a plurality of iron scraps. In one embodiment, the layer information may be information about the heights of a plurality of iron scraps that changes over time during the unloading process. That is, the layer information may include position change information and / or height change information about a plurality of iron scraps updated when one or more iron scraps are unloaded from the loading device. In one embodiment, the method of obtaining the layer information may include a single-type method using one camera or a stereo-type method using two or more cameras. This will be described with reference to Figures 3 to 5 This will be described.

[0054] Figure 3 FIG. is a diagram for describing an example in which the iron scrap sorting device 100 according to one embodiment of the present disclosure obtains layer information based on a single-type method.

[0055] Referring to Figure 3 , in the single-type method, depth information obtained by analyzing changes in the wall surface of the loading box obtained from an image acquired by one camera may be used. For example, the iron scrap sorting device 100 may obtain depth information indicating the heights of a plurality of iron scraps in the loading device based on the changes in the wall surface of the loading box according to the loading state images captured in the vertical direction and / or one or more lateral directions while one camera moves.

[0056] Figure 4 FIG. is a diagram for describing an example in which the iron scrap sorting device 100 according to one embodiment of the present disclosure obtains layer information based on a stereo-type method.

[0057] Referring to Figure 4, in the stereoscopic method, depth information obtained by analyzing the angular changes obtained from images of the same region can be used, and the images are obtained from two or more cameras located on the same plane. For example, the iron scrap sorting device 100 can obtain depth information indicating the heights of multiple iron scraps in the loading device based on the image changes caused by the angular changes of the same region in multiple loading state images captured in the vertical direction from two or more cameras located on the same plane. The iron scrap sorting device 100 can use an artificial intelligence (AI) model to obtain layer information based on the depth information obtained according to the single-type method and / or the stereoscopic method. That is, the iron scrap sorting device 100 can analyze the changes in the wall surface of the loading box using the single-type method, and can analyze the image changes according to the angular changes using the stereoscopic method. Therefore, the iron scrap sorting device 100 obtains layer information based on the depth information according to the changes in the wall surface of the loading box and the depth information according to the image changes of the region corresponding to the same region in the loading state image, and uses the layer information for image analysis, thereby improving the accuracy of the analysis. That is, the limitations such as light reflection and jitter that may occur in the single-type method using one camera can be supplemented by using the stereoscopic method of two or more cameras.

[0058] Figure 5 It is a diagram for describing an example of the iron scrap sorting device 100 using depth information according to an embodiment of the present disclosure.

[0059] Refer to Figure 5 , the loading state image shown in the upper part of the figure is an image obtained according to the single-type method, and the iron scrap sorting device 100 can obtain the loading state image captured in the vertical direction using an AI model. In addition, the loading state image captured in the horizontal direction can be further obtained. Therefore, the changes in the wall surface of the loading box can be analyzed based on the loading state images captured in the vertical and horizontal directions. For example, the region with large changes in the wall surface of the loading box can be a region with a shallow depth, and the region with small changes in the wall surface of the loading box can be a region with a deep depth. The iron scrap sorting device 100 can obtain a depth map as shown in the lower part of the figure, and the depth map shows the heights or height changes of multiple iron scraps. As Figure 5 shown, each of the multiple iron scrap regions can be represented by different colors or different shaded regions according to the depth. That is, the denser the shadow density, the deeper the region can be represented. Therefore, the iron scrap sorting device 100 can obtain layer information including layer changes using the depth information of the depth map.

[0060] Referring to operation S230, the iron waste sorting device 100 according to one embodiment may determine a region of interest based on a loaded state image updated as the unloading process progresses. In one embodiment, the updated loaded state image may be an image after one or more iron wastes have been unloaded as the unloading process progresses, and the unloading area may correspond to the region of interest. In one embodiment, the iron waste sorting device 100 may determine the region of interest based on a result of comparing a first loaded state image corresponding to a first time point with a second loaded state image corresponding to a second time point that is later in time than the first time point. For example, the first loaded state image may correspond to an image before one or more iron wastes are unloaded, and the second loaded state image may correspond to an image after one or more iron wastes are unloaded.

[0061] Figure 6 FIG. is an example for describing that the iron waste sorting device 100 according to one embodiment of the present disclosure determines a region of interest based on whether a grapple is included in any partial region of a vertical direction region of a loading device.

[0062] Referring to Figure 6 , in one embodiment, during the unloading process of one or more iron wastes, a partial region in the vertical direction of the loading device including the grapple may be determined as the region of interest. For example, a partial region (a region in the vertical direction) corresponding to a case where the grapple satisfies a condition of appearing or disappearing above the loading device may be determined as the region of interest.

[0063] Figure 7 FIG. is an example for describing that the iron waste sorting device 100 according to one embodiment of the present disclosure obtains a grapple operation region.

[0064] Referring to Figure 7 , the iron waste sorting device 100 may obtain a grapple operation region of an image including the grapple from a plurality of loaded state images captured during the unloading process. The iron waste sorting device 100 may identify the grapple from the loaded state image using an AI model. As Figure 7As shown, the grapple can be recognized, and the grapple operation area can be obtained. For example, the iron scrap sorting device 100 can obtain a specific area including the boundary of the grapple as the grapple operation area. Therefore, the iron scrap sorting device 100 can determine a partial area including the grapple operation area as the region of interest. The iron scrap sorting device 100 according to one embodiment can determine a partial area corresponding to the grapple operation area among the different areas between the first loading state image and the second loading state image as the region of interest. The iron scrap sorting device 100 can compare and analyze the image difference between the region of interest corresponding to the grapple operation area in the first loading state image and the region of interest corresponding to the grapple operation area in the second loading state image.

[0065] Referring to operation S240, the iron scrap sorting device 100 according to one embodiment can obtain a segmentation image of a target iron scrap that is included in the region of interest and is any one of a plurality of iron scraps. The iron scrap sorting device 100 can obtain a segmentation image including the target iron scrap from the loading state image using a segmentation model. In one embodiment, the target iron scrap can be the iron scrap that is the target of image analysis. In order to classify a plurality of iron scraps included in the region of interest into one iron scrap, the iron scrap sorting device 100 can perform segmentation on the iron scrap in any one of the plurality of iron scrap regions to obtain a segmentation image. In one embodiment, the segmentation image can include a plurality of images. For example, the iron scrap sorting device 100 can perform instance segmentation on the loading state image to obtain a segmentation image of a single object. In addition, the iron scrap sorting device 100 can perform semantic segmentation on the loading state image to obtain a segmentation image of an aggregated object. In one embodiment, semantic segmentation can be performed in units of pixels. For example, the iron scrap sorting device 100 can obtain a segmentation region including one or more pixels corresponding to each of the plurality of iron scraps in units of pixels to obtain a segmentation image, which is an image corresponding to each segmentation region obtained in the region of interest. Therefore, the segmentation image can be a multi-concept including both a segmentation image of a single object and a segmentation image of an aggregated object. Semantic segmentation is a segmentation performed by recognizing an object as a physical semantic unit that can actually be recognized and can obtain a segmentation image of an aggregated object in which a plurality of objects are clustered, while instance segmentation is a segmentation performed by recognizing each object as a single unit and can obtain a segmentation image of a single object.

[0066] Referring to operation S250, the iron scrap sorting device 100 according to one embodiment can obtain item information and grade information about the target iron scrap based on the segmented image and layer information. The iron scrap sorting device 100 can use a classification model that performs analysis on a per-image basis to obtain item information and grade information corresponding to the target iron scrap. In one embodiment, sorting is a process of analyzing or sorting iron scrap based on the feature information obtained from each image of each segmented image. The iron scrap sorting device 100 can use the classification model to perform analysis on each of the plurality of segmented images on a per-image basis. Thus, the iron scrap sorting device 100 can obtain item information and grade information corresponding to each analyzed image. In one embodiment, the item information corresponds to the feature information about the iron scrap and can include, for example, large-weight iron scrap information, small-weight iron scrap information, etc. Further, in one embodiment, the iron scrap sorting device 100 can perform sorting on the target iron scrap image to obtain grade information corresponding to the target iron scrap. That is, the iron scrap sorting device 100 can obtain item information and grade information corresponding to each target iron scrap by performing sorting. Further, the iron scrap sorting device 100 can use the layer information to perform sorting. In one embodiment, the sizes of the plurality of iron scraps in the loading state image can vary according to the layer. For example, for each of the plurality of iron scraps, the higher the loading height, the larger the size that may appear in the loading state image. In one embodiment, since the size of the iron scrap may be a factor affecting the item information or grade information, the iron scrap sorting device 100 can also use the depth information included in the layer information to obtain the item information and grade information by performing sorting on the segmented image. The depth information is a factor that can determine the height of each of the plurality of iron scraps and can be a factor indicating a more detailed height (depth) for determining the layer. The iron scrap sorting device 100 can use the depth information of each of the plurality of iron scraps included in the segmented image and the sorting result to obtain the item information and grade information of each of the plurality of iron scraps. Thus, the iron scrap sorting device 100 can more precisely determine the item information and grade information by considering the size of the iron scrap according to the height (depth), not only using the images obtained in a planar manner but also using the depth information, and thus can improve the accuracy of the item information and grade information of the iron scrap.

[0067] Referring to operation S260, the iron scrap sorting device 100 according to one embodiment can provide sorting information about the iron scrap including the item information and grade information. The iron scrap sorting device 100 can provide the item information and grade information of the target iron scrap obtained using the segmentation model and the classification model as described above as the sorting information about the iron scrap.

[0068] Figure 8FIG. is an example of a scrap iron sorting apparatus 100 according to an embodiment of the present disclosure obtaining layer information according to a single type method and obtaining an unloading area based on a grapple operation area.

[0069] Referring to Figure 8 , the scrap iron sorting apparatus 100 according to an embodiment may be based on Figure 7The region of interest or the difference region of each of the first loading state image and the second loading state image is obtained as the unloading region in the grapple operation region described. That is, the iron scrap sorting device 100 can obtain the difference region between the first loading state image representing the loading state image before unloading one or more iron scraps and the second loading state image representing the loading state image after unloading one or more iron scraps, measure the change in the difference region, identify the unloading region, and determine a more accurate region of interest. That is, the iron scrap sorting device 100 can identify the unloading region based on the grapple operation region and the difference region. Therefore, the iron scrap sorting device 100 can determine the quadrilateral region (unloading region B-box) including the unloading region determined along the boundary indicating the difference region as the final region of interest. Therefore, since the classification information about the iron scrap is obtained by obtaining the segmented image of the region of interest of the unloading region determined along the boundary indicating the difference region, the advantage is that the error rate according to the change in the surrounding region other than the unloading region can be reduced. The iron scrap sorting device 100 according to one embodiment can determine the region of interest based on the operation state of the grapple. The operation state of the grapple indicates whether the grapple includes one or more iron scraps when the grapple is included in any partial region of the region in the vertical direction of the loading device for a preset time period. For example, the iron scrap sorting device 100 can determine whether the unloading of the iron scrap has occurred based on the time period during which the grapple is included in the partial region of the region in the vertical direction of the loading device. When the time period during which the grapple is included in a specific region is greater than the first time period, the iron scrap sorting device 100 can determine that the unloading of one or more iron scraps has occurred. In one embodiment, the first time period can be a preset time period corresponding to the case where the unloading of the iron scrap has occurred during the unloading process. In addition, the partial region can include a plurality of regions that are updated as the grapple moves. Therefore, when the total time period of each time period during which the grapple is included in each of the plurality of partial regions is greater than or equal to the first time period, the iron scrap sorting device 100 can determine that the unloading of the iron scrap has occurred. In addition, the iron scrap sorting device 100 can determine the region having the longest time period for including the grapple among the plurality of regions as the unloading region. That is, in one embodiment, the first time period can be a time period used as a criterion for determining whether the unloading of the iron scrap has occurred. The iron scrap sorting device 100 according to one embodiment can obtain the operation state of the grapple. The operation state indicates whether the grapple includes one or more iron scraps in each of the plurality of loading state images captured during the unloading process when the grapple is included in a specific region for a second time period longer than the first time period by a preset time period or more (for example, half of the first time period). The case where the time period during which the grapple is included in the partial region is greater than or equal to the second time period can be a case where it can be determined that no unloading has occurred.For example, the second time period is a time period that is greater than or equal to the first time period by a preset time period or longer, and the case where the time period during which the grapple is included in the partial area is greater than or equal to the second time period corresponds to a high probability that the current operation of the grapple is being executed in real time. Therefore, the iron scrap sorting device 100 can obtain the operation state of the grapple and determine the region of interest based on the operation state. For example, by obtaining the operation state of the grapple, it is possible to identify, among a plurality of loading state images, a loading state image in which the grapple includes one or more iron scraps. The iron scrap sorting device 100 can determine a certain boundary area where the grapple is located at the time point when the operation state of the grapple first includes iron scraps as the region of interest, or determine a certain boundary area where the grapple is located at the time point when the operation state of the grapple last includes iron scraps as the region of interest. The case where the time period during which the grapple is included in the partial area is greater than or equal to the second time period may correspond to the case where the grapple stays in the area in the vertical direction of the loading device for a long time period. That is, the case where there is a loading state image in which the grapple includes one or more iron scraps may correspond to the case where the grapple changes the positions of a plurality of iron scraps, rather than the case where the grapple unloads a plurality of iron scraps from the loading device. Therefore, the iron scrap sorting device 100 can determine the region of interest based on the first time point and the last time point when the grapple includes iron scraps. The region of interest corresponding to the first time point can be analyzed similarly to the state where some iron scraps are unloaded, and the region of interest corresponding to the last time point can be analyzed similarly to the state where some iron scraps are loaded. Therefore, the iron scrap sorting device 100 can determine the region of interest of the area where the iron scraps are unloaded based on the case where the grapple stays in the area in the vertical direction of the loading device and then disappears, and determine the region of interest of the area where the iron scraps are moved based on the operation state of the grapple when the grapple stays in the area in the vertical direction of the loading device for the second time period or longer. Therefore, there is an effect that analysis and classification can be performed on each of a plurality of iron scraps that can be located according to various situations. In another embodiment, the iron scrap sorting device 100 can obtain an operation mode based on the operation state of the grapple. For example, when the time period during which the grapple is included in the partial area is greater than or equal to the second time period, the iron scrap sorting device 100 can analyze a plurality of continuously captured loading state images to determine whether the operation state of the grapple repeats between a state including iron scraps and a state not including iron scraps. When an operation mode in which the state including iron scraps and the state not including iron scraps repeatedly occur is obtained, the iron scrap sorting device 100 can differently determine the position and number of regions of interest based on the order positions of the state including iron scraps and the state not including iron scraps according to the number of repetitions and the passage of time.For example, since the case where the repetition count is 1 means that the positions of some of the iron scraps among the multiple iron scraps have been moved once, the iron scrap sorting device 100 can determine the number of regions of interest to be two, and determine the positions of the regions of interest as the region corresponding to the position of the grapple during the time period when the grapple first includes an iron scrap and the region corresponding to the position of the grapple during the time period when the grapple last includes an iron scrap. In addition, the size of the region of interest can be determined based on the change between the previous loading state image (the image before update) and the current loading state image (the image after update). When the repetition count is 1, the iron scrap sorting device 100 can update the region of interest based on the probability of overlap of the two regions of interest. For example, when the sequential positions of the state including the iron scrap and the state not including the iron scrap are less than a preset distance (e.g., a length less than 10% of the horizontal length of the loading device), the iron scrap sorting device 100 can perform a first image analysis by overlapping the two regions of interest and updating the two regions of interest to one region of interest, perform a second image analysis on the two regions of interest, and perform a third image analysis by subdividing the two regions of interest and updating the two regions of interest to three regions of interest. That is, when the sequential positions of the state including the iron scrap and the state not including the iron scrap are less than the preset distance, the iron scrap sorting device 100 can provide classification information about the iron scrap through the third image analysis process. Since the probability of overlap of the two regions of interest is high when the sequential positions are less than the preset distance, the two regions of interest can overlap and be determined as one region of interest, and the item information and grade information of each target iron scrap (which is any one of the multiple iron scraps) can be obtained based on the layer information. In addition, each of the two regions of interest can be analyzed to obtain the item information and grade information of each target iron scrap. In addition, each of the three regions of interest divided based on the distance between the centers of the grapple in the state including the iron scrap and the state not including the iron scrap and 1 / 3 of the length of the average value as the preset distance can be analyzed to obtain the item information and grade information of each target iron scrap. Therefore, by comparing and analyzing the analysis results of the third image analysis process, the analysis accuracy of each target iron scrap can be improved. In another embodiment, when the repetition count is two or more times, as described above, the iron scrap sorting device 100 can determine that the process of analyzing a state similar to the state of unloading some iron scraps for one region of interest and analyzing a state similar to the state of loading some iron scraps for another region of interest is not applicable. That is, when the repetition count is two or more times, the iron scrap sorting device 100 can determine that it is a case of mixing multiple iron scraps and does not correspond to the case of unloading and / or loading.Therefore, when the number of repetitions is two or more, the iron scrap sorting process can be terminated, and the iron scrap sorting process can be re-executed by resetting at the time point when the grapple is no longer included in some areas (when the grapple disappears).

[0070] Figure 9 FIG. is an example for describing layer information at each layer change time point obtained by the iron scrap sorting device 100 according to a three-dimensional method according to an embodiment of the present disclosure.

[0071] Referring to Figure 9 , the iron scrap sorting device 100 according to an embodiment can obtain the first layer change time points at which the heights of a plurality of iron scraps have changed by a critical height or more based on the depth information obtained according to the three-dimensional method. In addition, the iron scrap sorting device 100 can obtain the second layer change time points at which the volumes of a plurality of iron scraps have changed by a critical percentage or more based on the average volume of the loading device. Therefore, the iron scrap sorting device 100 can obtain layer information based on at least one of the first layer change time point and the second layer change time point. In one embodiment, the iron scrap sorting device 100 can perform a layer change measurement process. A layer can be a concept of a unit corresponding to an iron scrap layer that has changed after unloading at least one iron scrap using a grapple. For example, the case where iron scraps are completely loaded onto the loading device can be referred to as layer-3, and the case where iron scraps are loaded onto the loading device exceeding the height of the loading device can be referred to as layer-3.5. In addition, at the time point when the height of the iron scraps in the loading device changes to a critical height or more during the unloading process, the layer can be updated one by one to a smaller layer (for example, layer-2 or layer-1). In one embodiment, when determining the item information and grade information of the target iron scrap using the layer information determined based on the depth information obtained according to the three-dimensional method, the advantage is that it is less sensitive to changes such as movement, and there is time between layers, so that higher-level AI or machine learning can be applied. As Figure 9 shown, when a plurality of iron scraps are completely loaded onto the loading device, the iron scrap sorting device 100 can determine the layer as layer-3 and determine a lower layer to reduce the loading height. In Figure 9 layer-1, it can be seen that the heights of the plurality of loaded iron scraps are low and the volumes of the plurality of iron scraps are small, and in Figure 9In layer - 3, it can be seen that the height of multiple loaded iron scraps is high and the volume of multiple iron scraps is large. The iron scrap sorting device 100 can obtain a depth map of each layer to obtain depth information. In one embodiment, assuming that the average unloading time period is about 5 minutes (300 seconds), about 20 unloading operations are performed with a grapple, and there are approximately 3 layers. When determining item information and grade information in units of grapples, the item information and grade information should be inferred within 15 seconds (300 seconds / 20 times), and when determining item information and grade information in units of layers, the item information and grade information should be determined within 100 seconds (300 seconds / 3 layers). Therefore, the advantage is that higher - level AI or machine - learning applications are feasible. In one embodiment, the iron scrap sorting device 100 can determine the time point when the height of multiple iron scraps is reduced to a critical height (or greater than about 0.4 m) as the first - layer change time point. In addition, the iron scrap sorting device 100 can obtain average volume information about the loading device from an administrator account. The iron scrap sorting device 100 can, based on the obtained average volume information and based on the average volume of the loading device, determine the time point when the volume of multiple iron scraps is reduced to a critical percentage (about 33%) or more as the second - layer change time point. Therefore, the iron scrap sorting device 100 can determine the layer at any time point that is faster in chronological order based on the first - layer change time point and / or the second - layer change time point, or at the average time point of the first - layer change time point and the second - layer change time point. The iron scrap sorting device 100 can obtain item information and grade information of the target iron scrap by further using the depth information obtained from at least one determined layer when analyzing the segmented image. That is, the change between the corresponding updated images of each layer can be measured based on the layer change time point. In another embodiment, the iron scrap sorting device 100 can assign different weights to the first - layer change time point and the second - layer change time point. For example, the iron scrap sorting device 100 can use layer information, and since the importance of height (depth) can be higher than that of volume, the weights can be assigned to each of the first - layer change time point and the second - layer change time point at a ratio of 7:3. In addition, the iron scrap sorting device 100 can determine the ratio differently according to the initial loading state of multiple loaded iron scraps. For example, the importance of height (depth) can be determined differently according to the loading height based on the initial loading state.For example, when loading multiple iron scraps corresponding to 100% of the height of the loading device based on the initial loading state, the weights can be allocated to the first-layer change time point and the second-layer change time point at a ratio of 8:2. When loading multiple iron scraps corresponding to 90% or more of the height of the loading device, the weights can be allocated to the first-layer change time point and the second-layer change time point at a ratio of 7:3. When loading multiple iron scraps corresponding to 80% or more and less than 90% of the height of the loading device, the weights can be allocated to the first-layer change time point and the second-layer change time point at a ratio of 6:4. And when loading multiple iron scraps corresponding to less than 80% of the height of the loading device, the weights can be allocated to the first-layer change time point and the second-layer change time point at a ratio of 5:5. Therefore, since the average time point of the obtained layer is determined by updating the ratio of the allocated weights, the importance of volume relative to length can be increased according to the loading height of the loading device, and thus the efficiency of using layer information can be improved. In another embodiment, the iron scrap sorting device 100 can determine the importance of volume differently according to the size of the loading device. For example, when the total volume of the loading device is less than a preset volume, the sensitivity of the volume may be higher, so the weights can be allocated to the first-layer change time point and the second-layer change time point at a ratio of 3:7. In addition, when the total volume of the loading device is greater than a certain multiple (3 times) of the preset volume, the critical height and the critical percentage can be updated. For example, when the total volume of the loading device is greater than a certain multiple, even when unloading a large amount of iron scraps, the height may not decrease much, so the critical height can be reduced to a length corresponding to 80% of the conventional critical height (e.g., about 0.32 m), and the critical percentage can be increased to a percentage corresponding to 120% of the conventional critical percentage (e.g., about 39.6%). Therefore, since the sensitivity of height and volume can be adjusted according to the situation, the efficiency can be improved. Each of the above numbers and percentages can be flexibly changed according to the situation, and can also be changed and set by the administrator.

[0072] Figure 10 It is a diagram for describing an example in which the iron scrap sorting device 100 according to an embodiment of the present disclosure performs semantic segmentation and instance segmentation.

[0073] Refer to Figure 10, the iron scrap sorting device 100 according to an embodiment may perform instance segmentation that divides objects of the same type into the same area, and / or perform instance segmentation that divides objects of the same type into different areas, so as to segment the types and boundary lines of the objects in the image. That is, the type and position of the iron scrap can be obtained from the loaded state image according to the AI learning model, and the item information and grade information corresponding to each iron scrap can be obtained. The iron scrap sorting device 100 according to an embodiment may perform semantic segmentation and instance segmentation in parallel. That is, instance segmentation may be performed after semantic segmentation, or semantic segmentation may be performed after instance segmentation. For example, as Figure 10 shown, the iron scrap sorting device 100 may obtain a segmented image of a single object by performing semantic segmentation and then performing instance segmentation. Semantic segmentation divides the iron scrap into the same area when the iron scraps are of the same type, and instance segmentation divides each segmented image of the aggregated object on which semantic segmentation has been performed into different iron scrap areas one by one. As Figure 10 shown, the iron scrap sorting device 100 may perform semantic segmentation that marks (segments) multiple iron scraps into an aggregated object, and perform instance segmentation that marks (segments) multiple iron scraps into each single object. In addition, conversely, the iron scrap sorting device 100 may perform semantic segmentation on the area excluding the area corresponding to the single object after performing image segmentation of the single object by performing instance segmentation. Therefore, even in areas where instance segmentation has not been accurately performed, the accuracy can be further improved by obtaining more segmented images of the aggregated object. This will be described with reference to Figure 11 .

[0074] Figure 11 is a diagram for describing an example of the iron scrap sorting device 100 according to an embodiment of the present disclosure further obtaining a segmented image of an aggregated object in an area excluding the area corresponding to a single object.

[0075] Referring to Figure 11 , the iron scrap sorting device 100 according to an embodiment may obtain a segmented image of the aggregated object by performing semantic segmentation on the area excluding the area corresponding to the single object obtained by instance segmentation in Figure 10 . Therefore, after performing semantic segmentation, instance segmentation may be further performed on the area corresponding to the aggregated object. Therefore, as shown in the lower part of Figure 11 , even for the area excluding the area corresponding to the single object, the area of each of the multiple iron scraps can be further segmented into single objects. In Figure 11Among them, although examples of performing semantic segmentation and instance segmentation on multiple iron scraps throughout the loading device are described, the present disclosure is not limited thereto, and semantic segmentation and instance segmentation can equally be applied to regions of interest. As described above, the iron scrap classification device 100 uses semantic segmentation and instance segmentation in parallel to obtain a segmented image of the region of each of the multiple iron scraps, so that the accuracy of the item information and grade information of each target iron scrap can be improved.

[0076] According to one embodiment, by performing image analysis and image classification using a segmentation model and a classification model, the classification accuracy of iron scraps can be improved by obtaining segmented images of each individual object and aggregated objects. In addition, the accuracy of image classification and analysis of the unloading area can be improved by obtaining layer information based on a single-type method and a three-dimensional method, and the accuracy of detecting the unloading area can be improved by determining the region of interest based on the grapple operation area. In addition, by using semantic segmentation and instance segmentation in parallel to obtain a segmented image, regions where each segmentation has not been accurately performed can be compensated for.

[0077] Various embodiments of the present disclosure can be implemented as software including one or more instructions stored in a storage medium (e.g., a memory) readable by a machine (e.g., a display device or a computer). For example, a processor (e.g., processor 120) of the machine can call at least one of the stored instructions from the storage medium and execute the instructions. This enables the device to operate to perform at least one function according to at least one called instruction. The one or more instructions can include code generated by a compiler or code executable by an interpreter. The storage medium readable by the device can be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), and this term does not distinguish between the case of semi-permanently storing data in the storage medium and the case of temporarily storing data.

[0078] According to one embodiment, the method according to various embodiments disclosed in the present disclosure can be included in a computer program product and provided. The computer program product can be traded between a seller and a buyer as a commodity. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or can be distributed online (e.g., by downloading or uploading) through an application store (e.g., Play Store TM), or directly between two user devices (e.g., a smart phone). In the case of online distribution, at least a part of the computer program product can be temporarily stored or temporarily generated in a machine-readable storage medium such as the memory of a manufacturer's server, an application store's server, or an intermediate server.

[0079] According to an embodiment of the present disclosure, by performing image analysis and image classification using a segmentation model and a classification model, the classification accuracy of iron scrap can be improved by obtaining segmentation images of each individual object and aggregated objects.

[0080] In addition, the accuracy of image classification and analysis of the unloading area can be improved by obtaining layer information based on a single-type method and a three-dimensional method.

[0081] In addition, by determining the region of interest based on the grapple operation area, the accuracy of detecting the unloading area can be improved.

[0082] In addition, by obtaining segmentation images by using semantic segmentation and instance segmentation in parallel, regions where each segmentation has not been accurately performed can be compensated.

[0083] The effects of the present disclosure are not limited to the above effects, and other effects not described can be clearly understood by those skilled in the art from the above detailed description.

[0084] Although the present disclosure has been described with reference to the accompanying drawings, the present disclosure is not limited to the disclosed embodiments and drawings, and those skilled in the art will understand that various changes can be made in form and detail without departing from the spirit and scope of the present disclosure. Therefore, the disclosed method should be considered from an exemplary perspective for description rather than a restrictive perspective. Even if embodiments are described and the effects of the configuration according to the present disclosure are not explicitly described, effects predictable from the configuration can be identified. The scope of the present disclosure is not limited by the detailed description of the present disclosure, but is defined by the appended claims, and encompasses all modifications and equivalents that fall within the scope of the appended claims and will be construed as being included in the present disclosure.

[0085] Cross-reference to related applications

[0086] This application claims the priority and benefits of Korean Patent Application No. 10-2023-0188834, filed on December 21, 2023, the disclosure of which is incorporated herein by reference in its entirety.

Claims

1. A method for providing classified information about iron scrap according to an unloading process, the method comprising the following steps: obtaining, by a receiving unit, a loading state image captured during an unloading process of a plurality of iron scraps loaded onto a loading device; obtaining, by a processor, layer information, the layer information being updated as the unloading process proceeds and being determined according to the heights of the plurality of scrap irons; determining, by the processor, a region of interest based on the loading status image updated as the unloading process proceeds; obtaining, by the processor, a segmented image of a target iron scrap, the target iron scrap being included in the region of interest and being any one of the plurality of iron scraps; obtaining, by the processor, item information and grade information about the target iron scrap based on the segmented image and the layer information; as well as Classification information including the item information and the grade information on the iron scrap is provided by the processor.

2. The method according to claim 1, wherein: A method of obtaining the layer information includes a single type method using one camera or a stereo type method using two or more cameras, and In the single type method, depth information obtained by analyzing a change in the wall surface of the loading box obtained from the image obtained by the one camera is used.

3. The method according to claim 1, wherein: The step of determining the region of interest comprises: The region of interest is determined by the processor based on a result of comparing a first loading status image corresponding to a first time point with a second loading status image corresponding to a second time point later in time than the first time point.

4. The method according to claim 3, wherein: The step of determining the region of interest further comprises: The region of interest is determined by the processor based on a difference area between the first loading status image and the second loading status image and an operation area of ​​a grapple used in the unloading process.

5. The method according to claim 1, wherein: The step of obtaining the segmented image by the processor includes: obtaining the segmented image including the target iron scrap from the loading state image using a segmentation model, and The step of obtaining, by the processor, the item information and the grade information includes obtaining the item information and the grade information corresponding to the target iron scrap using a classification model that performs analysis in units of images.

6. The method according to claim 5, wherein: The step of obtaining the segmented image comprises: performing instance segmentation on the loading state image and obtaining a segmented image of a single object by the processor; and The processor performs semantic segmentation on the loading state image and obtains a segmented image of the aggregated object.

7. The method according to claim 6, wherein: The operation of obtaining the segmented image of the aggregate object is performed on an area of ​​the loading state image other than an area corresponding to the single object excluded therefrom.

8. The method according to claim 2, wherein: In the stereoscopic type method, the depth information obtained by analyzing the angle change obtained from the images for the same area, the images being obtained from the two or more cameras located on the same plane, is used.

9. The method according to claim 4, wherein: The second time point is determined based on whether the grab hook is included in any partial area of ​​the area in the vertical direction of the loading device after the first time point; and The determining of the region of interest includes determining, by the processor, the region of interest based on an operation state of the grapple indicating whether the grapple includes at least one iron scrap when the grapple is included in the partial region for a preset period of time or longer.

10. The method according to claim 8, wherein: The method for obtaining the layer information comprises: obtaining, by the processor, a first layer change time point at which the height of the plurality of iron scraps changes to a critical height or higher based on the depth information obtained according to the stereoscopic method; obtaining, by the processor, a second level change time point at which the volume of the plurality of iron scraps changes to a critical percentage or more based on the average volume of the loading device; and The processor obtains the layer information at at least one of the first layer change time point and the second layer change time point.

11. An iron scrap sorting device for providing sorting information on iron scrap according to an unloading process, the iron scrap sorting device comprising: a receiving unit configured to obtain a loading state image captured during an unloading process of a plurality of iron scraps loaded onto a loading device; as well as A processor, the processor being configured to: obtaining layer information, wherein the layer information is updated as the unloading process proceeds and is determined according to the heights of the plurality of scrap iron; determining a region of interest based on the loading status image updated as the unloading process proceeds; obtaining a segmented image of a target iron scrap, the target iron scrap being included in the region of interest and being any one of the plurality of iron scraps; obtaining item information and grade information about the target iron scrap based on the segmented image and the layer information; and Classification information including the item information and the grade information on the iron scrap is provided.

12. The iron scrap sorting equipment according to claim 11, wherein: A method of obtaining the layer information includes a single type method using one camera or a stereo type method using two or more cameras, and In the single type method, depth information obtained by analyzing a change in the wall surface of the loading box obtained from the image obtained by the one camera is used.

13. The iron scrap sorting equipment according to claim 11, wherein: The processor determines the region of interest based on a result of comparing a first loading status image corresponding to a first time point with a second loading status image corresponding to a second time point later in time than the first time point.

14. The iron scrap sorting equipment according to claim 13, wherein: The processor determines the region of interest based on a difference area between the first loading status image and the second loading status image and an operation area of ​​a grapple used in the unloading process. 15 . A computer-readable recording medium having recorded thereon a program for executing the method according to claim 1 on a computer.

Images