Method and apparatus for providing iron scrap classification information based on the cargo handling process

Abstract

To improve accuracy of image analysis and classification for one or more iron scraps included in a loaded state image, in a loading process.SOLUTION: A method for providing iron scrap classification information by a loading process includes: allowing a reception part 110 to acquire a loaded state image imaged in a loading process for a plurality of iron scraps loaded on a loaded facility; allowing a processor 120 to acquire layer information determined by the height of the plurality of iron scraps and determine a region of interest, with progress of the loading process; acquiring a divided image for an object iron scrap included in the region of interest; acquiring item information and grade information for the object iron scrap; and providing iron scrap classification information including the item information and the grade information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The technical field of the present disclosure relates to a method for providing image classification result information in a cargo handling process of an article loaded on a loading equipment or a single article, etc., and relates to a technical field of a method for providing iron scrap classification information by image analysis for an area where one or more iron scraps are unloaded.

Background Art

[0002] Due to the recent increase in the logistics industry, in fact, loading equipment for loading various articles or loading articles to move them from one place to another is widely used. However, although a plurality of articles loaded on the loading equipment can be managed or loaded according to the product type and the position by specification by the judgment of the operator, it may be difficult to confirm this every time. Therefore, a segmentation method may be used to monitor the images of each area by dividing the entire area of the loading equipment on which a plurality of articles are loaded. Generally, in the case of a segmentation method for iron scrap-related image analysis, the optical system (CCTV and mechanical parts) is easily installed at the cargo handling (unloading) location without H / W engineering to set the position and angle so that AI can exhibit optimal performance, and images are acquired. In such a case, there are few consistent features (various shapes by utilization method, plain texture (texture), various color sensations (painting, rust, etc.)), and an irregular iron scrap due to various cutting methods, warping, etc. The method of determining the grade of iron scrap by classifying it by instance segmentation is adopted, so there is a limitation that a large amount of image collection and labeling are required. Therefore, in fact, there is a need to provide a service that can improve performance with less image collection and labeling by providing an image analysis method and a system that can solve the limitation of such an analysis method that requires a large amount of image collection.

Prior Art Documents

Patent Documents

[0003] [Patent Document 1] Korean Published Patent No. 10-2011-0078566 (2011.07.07) Efficient Item Loading Position Detection System Using Digital Image Recognition [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] The problem that this disclosure aims to solve is to improve the accuracy of image analysis and classification of one or more pieces of scrap iron included in a loading state image when providing classification information for scrap iron during the cargo handling process, and to provide a service that acquires segmented images of areas determined to be unloading areas from loading state images captured during the cargo handling process, and provides highly accurate image classification information for the segmented images.

[0005] The issues that this disclosure aims to solve are not limited to the technical issues described above, and other technical issues may also exist. [Means for solving the problem]

[0006] As a technical means for achieving the aforementioned technical challenges, a method for providing scrap iron classification information by a cargo handling process relating to the first aspect of this disclosure includes the steps of: a receiving unit acquiring a loading state image of a plurality of scrap iron loaded on a loading device during the cargo handling process; a processor acquiring layer information which is updated as the cargo handling process progresses and is determined by the height of the plurality of scrap iron; a processor determining a region of interest based on the loading state image which is updated as the cargo handling process progresses; a processor acquiring a segmented image of a target scrap iron which is included in the region of interest and is one of the plurality of scrap iron; a processor acquiring item information and grade information for the target scrap iron based on the segmented image and the layer information; and a processor providing scrap iron classification information which includes the item information and grade information.

[0007] Furthermore, the method for acquiring the layer information includes a monotype using one camera or a stereotype using two or more cameras, and the monotype can utilize depth information acquired through analysis of changes in the loading platform wall surface acquired from the image acquired by the one camera.

[0008] Furthermore, the step of determining the region of interest can be performed by the processor based on a comparison result between a first loading state image corresponding to a first time point and a second loading state image corresponding to a second time point that occurs later in time than the first time point.

[0009] Furthermore, in the step of determining the region of interest, the processor can determine the region of interest based on the difference region between the first loading state image and the second loading state image and the operating region of the grapple used in the cargo handling process.

[0010] Furthermore, in the stage where the processor acquires the segmented image, it can use a segmentation model to acquire the segmented image including the target iron scrap from the loading state image, and in the stage where the processor acquires the item information and grade information, it can use a classification model that performs analysis on an image-by-image basis to acquire the item information and grade information corresponding to the target iron scrap.

[0011] Furthermore, the step of acquiring the segmented image may include the step of the processor performing instance segmentation on the loading state image to acquire the segmented image for a single object; and the step of the processor performing semantic segmentation on the loading state image to acquire the segmented image for a group of objects.

[0012] Furthermore, the step of acquiring the segmented image of the crowd of objects can be performed on the region from which the region corresponding to the single object has been excluded from the loading state image.

[0013] Furthermore, the stereotype can utilize depth information obtained through analysis of angular changes acquired from images acquired from two or more cameras located on the same plane for the same region.

[0014] Furthermore, the second time point is determined based on whether or not the grapple is included in any part of the vertical region of the loading equipment after the first time point, and the step of determining the region of interest is, if the grapple is included in the part of the region for a predetermined time or longer, the processor can determine the region of interest based on the operating state of the grapple regarding whether the grapple contains at least one or more pieces of scrap iron.

[0015] Furthermore, the method for acquiring the layer information allows the processor to acquire a first layer change point when the height of the multiple scrap iron pieces changes to a critical length or more based on the depth information acquired by the stereotype, a second layer change point when the volume of the multiple scrap iron pieces changes to a critical percentage or more based on the average volume of the loaded equipment, and the processor to acquire the layer information based on at least one of the first layer change point and the second layer change point.

[0016] A scrap iron classification device that provides scrap iron classification information based on a cargo handling process relating to the second aspect of this disclosure may include: a receiving unit that acquires a loading state image of a plurality of scrap irons loaded on a loading device that is captured during the cargo handling process; and a processor that acquires layer information which is updated as the cargo handling process progresses and is determined by the height of the plurality of scrap irons, determines a region of interest based on the loading state image which is updated as the cargo handling process progresses, acquires a segmented image of a target scrap iron that is included in the region of interest and is one of the plurality of scrap irons, acquires item information and grade information for the target scrap iron based on the segmented image and the layer information, and provides scrap iron classification information including the item information and the grade information.

[0017] Furthermore, the method for acquiring the layer information includes a monotype using one camera or a stereotype using two or more cameras, and the monotype can utilize depth information acquired through analysis of changes in the loading platform wall surface acquired from the image acquired by the one camera.

[0018] Furthermore, the processor can determine the region of interest based on a comparison result between a first loading state image corresponding to a first time point and a second loading state image corresponding to a second time point that occurs later in time than the first time point.

[0019] Further, the processor can determine the region of interest based on the difference region between the first loading state image and the second loading state image and the operation region of the grapple used in the handling process.

[0020] According to a third aspect of the present disclosure, a computer-readable non-transitory recording medium on which a program for implementing the method of the first aspect is recorded can be provided.

Effects of the Invention

[0021] According to an embodiment of the present disclosure, there is an effect that the accuracy of scrap classification can be improved by obtaining segmented images for single objects and clustered objects, respectively, in terms of performing image analysis and image classification using a segmentation model and a classification model.

[0022] Further, in terms of obtaining layer information based on monotype and stereotype, there is an effect that the accuracy of image classification and analysis for the unloading area can be increased.

[0023] Further, in terms of determining the region of interest based on the grapple operation region, there is an effect that the accuracy of detecting the unloading area can be increased.

[0024] Further, in terms of obtaining a segmented image by using semantic segmentation and instance segmentation in parallel, there is an advantage that it is possible to complement regions where each segmentation is not accurately performed.

[0025] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

Brief Description of the Drawings

[0026] [Figure 1]This is a block diagram schematically illustrating the configuration of an iron scrap classification device that provides iron scrap classification information according to one embodiment of the present disclosure. [Figure 2] This flowchart shows the steps involved in providing iron scrap classification information during the handling process for an iron scrap classification device according to one embodiment of the present disclosure. [Figure 3] This drawing illustrates an example of how an iron scrap sorting apparatus according to one embodiment of the present disclosure acquires layer information based on monotype. [Figure 4] This is a drawing illustrating an example of how an iron scrap sorting apparatus according to one embodiment of the present disclosure acquires layer information based on stereotypes. [Figure 5] This drawing illustrates an example of how an iron scrap sorting device according to one embodiment of the present disclosure utilizes depth information. [Figure 6] This drawing illustrates an example of a scrap iron sorting device according to one embodiment of the present disclosure that determines a region of interest based on whether or not the grapple is included in any part of the vertical region of the loading equipment. [Figure 7] This drawing illustrates an example of how an iron scrap sorting device according to one embodiment of the present disclosure acquires a grapple operating range. [Figure 8] This drawing illustrates an example of an iron scrap sorting device according to one embodiment of the present disclosure, which acquires layer information using a monotype and acquires an unloading area based on the grapple operating area. [Figure 9] This drawing illustrates an example of how an iron scrap sorting apparatus according to one embodiment of the present disclosure acquires layer information by layer change point using stereotypes. [Figure 10] This diagram schematically illustrates an example of how an iron scrap sorting apparatus according to one embodiment of the present disclosure performs semantic segmentation and instance segmentation. [Figure 11] This diagram schematically illustrates an example of how an iron scrap sorting apparatus according to one embodiment of the present disclosure further acquires a segmented image for a group of objects in a region where the region corresponding to a single object has been excluded. [Modes for carrying out the invention]

[0027] The advantages and features of this disclosure, and how they are achieved, will become clearer with reference to the embodiments described below in detail with the accompanying drawings. However, this disclosure is not limited to the embodiments disclosed below and may be embodied in a variety of different forms, although these embodiments are provided to complete the disclosure and to fully inform a person of the ordinary skill in the art of the subject of this disclosure of its scope.

[0028] The terms used herein are for illustrative purposes only and are not intended to limit the disclosure. In this specification, singular terms include plural terms unless otherwise specified in the text. The terms “comprises” and / or “comprising” used in the specification do not exclude the existence or addition of one or more other components in addition to the components mentioned. Throughout the specification, the same reference numerals refer to the same component, and “and / or” includes each of the components mentioned and all combinations of one or more of them. Although terms such as “first,” “second,” etc., are used to describe a variety of components, these components are not limited by these terms. These terms are simply used to distinguish one component from another. Therefore, the first component mentioned below may, of course, be the second component within the technical concept of this disclosure.

[0029] Unless otherwise defined, all terms used herein (including technical and scientific terms) should be used in a sense that can be commonly understood by an ordinary technician in the relevant technical field. Furthermore, terms defined in commonly used dictionaries should not be interpreted ideally or excessively unless explicitly defined otherwise.

[0030] Spatially relative terms such as "below," "beneath," "lower," "above," and "upper" can be used to easily describe the correlation between one component and another, as illustrated in the drawing. Spatially relative terms should be understood to include not only the directions illustrated in the drawing but also the different directions of components during use or operation. For example, if a component illustrated in the drawing is flipped over, a component described as "below" or "beneath" of another component may be placed "above" of that component. Therefore, the exemplary term "below" can include both the downward and upward directions. Components can also be oriented in other directions, and accordingly, spatially relative terms can be interpreted in terms of orientation.

[0031] The following describes an embodiment in detail with reference to the drawings.

[0032] Figure 1 is a block diagram schematically illustrating the configuration of an iron scrap sorting device 100 through image analysis according to one embodiment of the present disclosure.

[0033] Referring to Figure 1, the iron scrap sorting device 100 can include a receiving unit 110 and a processor 120.

[0034] According to one embodiment, the receiving unit 110 can acquire images of the loading state of multiple pieces of scrap iron loaded on the loading equipment, which are captured during the loading and unloading process.

[0035] In one embodiment, the processor 120 is updated as the handling process progresses and can acquire layer information determined by the heights of multiple pieces of scrap iron. The processor 120 can also determine a region of interest based on the loading state image, which is updated as the handling process progresses. The processor 120 can also acquire a segmented image of a target piece of scrap iron that is included in the region of interest and is one of the multiple pieces of scrap iron. The processor 120 can also acquire item information and grade information for the target piece of scrap iron based on the segmented image and layer information. Furthermore, the processor 120 can provide scrap iron classification information, including item information and grade information.

[0036] Furthermore, it should be noted that the scrap iron sorting device 100, which provides scrap iron sorting information based on the handling process, acquires a loading state image with a receiving unit 110, acquires layer information based on the heights of multiple scrap iron pieces with a processor 120, determines areas of interest based on the loading state image to acquire segmented images for the scrap iron, and provides scrap iron sorting information for the segmented images. This process can be connected by various conventional network combinations such as the Internet or mobile communication networks, and there are no special restrictions on this.

[0037] In addition to the components shown in Figure 1, a person with ordinary skill in the relevant art will understand that other general-purpose components may be further included in the scrap iron sorting device 100 that provides scrap iron sorting information by handling process through image analysis. For example, the scrap iron sorting device 100 that provides scrap iron sorting information by handling process may further include a memory (not shown) that stores loading state images, layer information, regions of interest, segmented images, item information and grade information corresponding to the segmented images, and may further include a transmission unit (not shown) that provides scrap iron sorting information or a display unit (not shown) that displays scrap iron sorting information. Alternatively, a person with ordinary skill in the relevant art will understand that some of the components shown in Figure 1 may be omitted if other embodiments are followed.

[0038] The iron scrap sorting device 100, which provides iron scrap sorting information based on the handling process according to one embodiment, can be used by users and can be linked with all types of handheld wireless communication devices equipped with a touchscreen panel, such as mobile phones, smartphones, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), and tablet PCs. In addition, it can be included in or linked with devices that have a platform on which applications can be installed and run, such as desktop PCs, tablet PCs, laptop PCs, and IPTVs including set-top boxes.

[0039] The scrap iron sorting device 100, which provides scrap iron sorting information based on the handling process, can be implemented as a terminal such as a computer that operates through a computer program to implement the functions described herein.

[0040] The iron scrap classification device 100 that provides iron scrap classification information based on the handling process according to one embodiment may include, but is not limited to, a system (not shown) that provides classification information for iron scrap and an associated server (not shown). The server according to one embodiment can support an application that provides a service for providing classification information for iron scrap.

[0041] The following description will focus on an embodiment in which a scrap iron classification device 100, which provides scrap iron classification information based on a cargo handling process according to one embodiment, independently acquires and provides classification information results using a pre-set scrap iron classification method. However, as mentioned above, this may also be performed through linkage with a server. In other words, the scrap iron classification device 100 that provides scrap iron classification information based on a cargo handling process according to one embodiment and the server can be integrated and implemented in terms of their functions, the server may be omitted, and it can be seen that the embodiment is not limited to just one example.

[0042] In one embodiment, the iron scrap sorting device 100 and the server may be linked, and the configuration for providing iron scrap classification information by performing the iron scrap classification process and the classification result information provision process may be performed by the server or by the iron scrap sorting device 100. For example, the iron scrap sorting device 100 can operate as a server, and in the following description, we will refer to it simply as the iron scrap sorting device 100.

[0043] Figure 2 is a flowchart showing the steps in which the iron scrap sorting device 100 according to one embodiment of the present disclosure provides iron scrap sorting information during the handling process.

[0044] Referring to step S210, the iron scrap sorting device 100 according to one embodiment can acquire a loading state image of multiple iron scraps captured during the handling process of the loading equipment. In one embodiment, the iron scrap sorting device 100 can acquire a loading state image captured from the upper side of the loading equipment downwards, at which time the loading equipment may be loaded with multiple iron scraps. Furthermore, the loading state image is an image acquired during the handling process and may include images after at least one or more iron scraps have been unloaded and / or images before the iron scraps have been unloaded as time progresses during the handling process. Therefore, the iron scrap sorting device 100 can acquire a loading state image containing multiple iron scraps during the handling process of at least one or more iron scraps.

[0045] Referring to step S220, the iron scrap sorting device 100 according to one embodiment is updated as the handling process progresses and can acquire layer information determined by the heights of multiple iron scraps. In one embodiment, the layer information may be information based on the heights of multiple iron scraps that change over time during the handling process. That is, the layer information may include position change information and / or height change information of multiple iron scraps that is updated when at least one or more iron scraps are unloaded from the loading equipment. In one embodiment, the method for acquiring the layer information may include a monotype using one camera or a stereotype using two or more cameras. This can be explained with reference to Figures 3 to 5.

[0046] Figure 3 is a diagram illustrating an example of how a scrap iron sorting device 100 according to one embodiment of the present disclosure acquires layer information based on monotype.

[0047] Referring to Figure 3, the monotype can utilize depth information obtained through analysis of changes in the loading platform wall surface acquired from images captured by a single camera. For example, the iron scrap sorting device 100 can acquire depth information indicating the height of multiple iron scraps in the loading equipment based on changes in the loading platform wall surface, based on loading state images captured in the vertical and / or one or more lateral directions as a single camera moves.

[0048] Figure 4 is a diagram illustrating an example of how a scrap iron sorting device 100 according to one embodiment of the present disclosure acquires layer information based on stereotypes.

[0049] Referring to Figure 4, stereotypes can utilize depth information obtained through analysis of angular changes acquired from images acquired from two or more cameras located on the same plane for the same region. For example, the scrap iron sorting device 100 can acquire depth information indicating the height of multiple scrap iron pieces in the loading equipment by analyzing image changes due to angular changes for the same region in multiple loading state images captured vertically from two or more cameras located on the same plane. The scrap iron sorting device 100 can acquire layer information based on depth information acquired using an AI model, monotype, and / or stereotype. That is, the scrap iron sorting device 100 may use monotype to perform analysis of changes in the loading platform wall surface, or use stereotype to perform analysis of image changes due to angular changes. Therefore, the scrap iron sorting device 100 can acquire layer information based on depth information due to changes in the loading platform wall surface and depth information due to image changes in regions corresponding to the same region in the loading state images, and use this for image analysis, which can improve the accuracy of the analysis. In other words, the limitations of monotype imaging, such as light reflection and shaking, which can occur with monotype imaging using a single camera, can be compensated for by using stereotype imaging with two or more cameras.

[0050] Figure 5 is a diagram illustrating an example in which an iron scrap sorting device 100 according to one embodiment of the present disclosure utilizes depth information.

[0051] Referring to Figure 5, the loading state image shown at the top is an image acquired by Monotype, and the iron scrap sorting device 100 can acquire a loading state image captured in the vertical direction using an AI model. Furthermore, it can acquire a loading state image captured in the lateral direction. Therefore, based on the loading state images captured in the vertical and lateral directions, it is possible to perform an analysis of changes in the loading platform wall surface. For example, the greater the change in the loading platform wall surface, the shallower the region may be, and the smaller the change in the loading platform wall surface, the deeper the region may be. The iron scrap sorting device 100 can acquire a depth map showing the height or height change of multiple iron scraps shown at the bottom. As shown in Figure 5, each of the multiple iron scrap regions may be shown with different hues or different hatching regions depending on the depth. That is, the finer the hatching density, the deeper the region. Therefore, the iron scrap sorting device 100 can acquire layer information including layer changes using the depth information of the depth map.

[0052] Referring to step S230, the iron scrap sorting device 100 according to one embodiment can determine a region of interest based on a loading state image that is updated as the handling process progresses. In one embodiment, the updated loading state image may be an image after at least one or more pieces of iron scrap have been unloaded as the handling process progresses, and the unloading region may correspond to the region of interest. In one embodiment, the iron scrap sorting device 100 can determine a region of interest based on a comparison result between a first loading state image corresponding to a first time point in time and a second loading state image corresponding to a second time point that occurs later in time than the first time point. For example, the first loading state image may correspond to an image before at least one or more pieces of iron scrap have been unloaded, and the second loading state image may correspond to an image after at least one or more pieces of iron scrap have been unloaded.

[0053] Figure 6 is a diagram illustrating an example in which an iron scrap sorting device 100 according to one embodiment of the present disclosure determines a region of interest based on whether or not the grapple is included in any part of the vertical region of the loading equipment.

[0054] Referring to Figure 6, in one embodiment, a portion of the vertical region of the loading equipment containing the grapple can be determined as the region of interest during the handling process of one or more pieces of iron scrap. For example, a portion of the region corresponding to the condition that the grapple appears and disappears on the loading equipment (vertical region) can be determined as the region of interest.

[0055] Figure 7 is a diagram illustrating an example of how an iron scrap sorting device 100 according to one embodiment of the present disclosure acquires a grapple operating range.

[0056] Referring to Figure 7, the scrap iron sorting device 100 can acquire the grapple operation region of an image containing a grapple from multiple loading state images captured during the loading process. The scrap iron sorting device 100 can identify grapples in the loading state images using an AI model. As illustrated in Figure 7, a grapple can be identified and a grapple operation region can be acquired. For example, the scrap iron sorting device 100 can acquire a certain area containing the grapple boundary as the grapple operation region. Therefore, the scrap iron sorting device 100 can determine a portion of the area including the grapple operation region as a region of interest. In one embodiment, the scrap iron sorting device 100 can determine a portion of the difference region between the first loading state image and the second loading state image that corresponds to the grapple operation region as a region of interest. The scrap iron sorting device 100 can compare and analyze the image difference between the region of interest corresponding to the grapple operation region in the first loading state image and the region of interest corresponding to the grapple operation region in the second loading state image.

[0057] Referring to step S240, the scrap iron sorting device 100 according to one embodiment can acquire a segmented image for a target scrap iron that is one of several scrap iron pieces included in the region of interest. The scrap iron sorting device 100 can acquire a segmented image that includes the target scrap iron in the loading state image by utilizing a segmentation model. In one embodiment, the target scrap iron may be the scrap iron that is the subject of image analysis. In order to classify the several scrap iron pieces included in the region of interest into individual scrap iron pieces, the scrap iron sorting device 100 can acquire a segmented image by performing segmentation for the scrap iron in one of several scrap iron regions. In one embodiment, the segmented image may include multiple images. For example, the scrap iron sorting device 100 can acquire a segmented image for a single object by performing instance segmentation on the loading state image. Alternatively, the scrap iron sorting device 100 can acquire a segmented image for a group of objects by performing semantic segmentation on the loading state image. In one embodiment, semantic segmentation can be performed at the pixel level. For example, the iron scrap sorting device 100 can acquire segmented regions containing at least one pixel corresponding to each of several iron scraps based on the pixel level, and acquire segmented images which are images corresponding to each segmented region acquired in the region of interest. Therefore, segmented images can be multiple concepts that include segmented images for single objects and segmented images for crowd objects. Semantic segmentation performs segmentation by recognizing objects as physically meaningful units that can actually be recognized, and segmented images for crowd objects formed by the aggregation of multiple objects can be acquired, while instance segmentation performs segmentation by recognizing each object as a single unit, and segmented images for single objects can be acquired.

[0058] Referring to step S250, the iron scrap classification device 100 according to one embodiment can acquire item information and grade information for target iron scrap based on segmented images and layer information. The iron scrap classification device 100 can acquire item information and grade information corresponding to target iron scrap by utilizing a classification model that performs analysis on an image-by-image basis. In one embodiment, classification may be a process of analyzing or classifying iron scrap for each segmented image based on the characteristic information acquired from each image. The iron scrap classification device 100 can perform analysis on an image-by-image basis for each of multiple segmented images using the classification model. Therefore, the iron scrap classification device 100 can acquire item information and grade information corresponding to each analysis image. In one embodiment, the item information corresponds to characteristic information for iron scrap and may include, for example, weight information and lightweight information. Also, in one embodiment, the iron scrap classification device 100 can perform classification on target iron scrap images to acquire grade information corresponding to target iron scrap. In other words, the scrap iron sorting device 100 can acquire item information and grade information corresponding to each target scrap iron by performing classification. Furthermore, the scrap iron sorting device 100 can utilize layer information when performing classification. In one embodiment, the size of each of the multiple scrap iron pieces in the loading state image may be shown differently depending on the layer. For example, the higher the stacking height of each of the multiple scrap iron pieces, the larger their size may be shown in the loading state image. In one embodiment, since the size of the scrap iron can be an element that affects the item information or grade information, the scrap iron sorting device 100 can utilize depth information included in the layer information when performing classification on the divided image to acquire item information and grade information. Depth information is an element that allows confirmation of the height of each of the multiple scrap iron pieces and can be an element that indicates a more detailed height (depth) for determining the layer.The scrap iron sorting device 100 can acquire item information and grade information for each of the multiple scrap iron pieces by utilizing the classification results and the depth information of each of the multiple scrap iron pieces included in the divided image. Therefore, the scrap iron sorting device 100 can determine the item information and grade information in more detail by further utilizing the depth information to consider the size of the scrap iron piece based on its height (depth), in addition to the image acquired in a planar manner, thus potentially increasing the accuracy of the item information and grade information for the scrap iron pieces.

[0059] Referring to step S260, the iron scrap classification device 100 according to one embodiment can provide iron scrap classification information including item information and grade information. As described above, the iron scrap classification device 100 can provide item information and grade information for target iron scrap obtained using a segmentation model and a classification model in the iron scrap classification information.

[0060] Figure 8 is a diagram illustrating an example in which an iron scrap sorting device 100 according to one embodiment of the present disclosure acquires layer information using monotype and acquires an unloading area based on the grapple operating area.

[0061] Referring to Figure 8, the iron scrap sorting device 100 according to one embodiment can acquire the region of interest or difference region of the first loading state image and the second loading state image as the unloading region based on the grapple operating region described in Figure 7. That is, the iron scrap sorting device 100 can acquire the difference region between the first loading state image, which shows the loading state image before at least one or more iron scraps are unloaded, and the second loading state image, which shows the loading state image after at least one or more iron scraps are unloaded, measure the change in the difference region to identify the unloading region, and determine a more accurate region of interest. In other words, the iron scrap sorting device 100 can identify the unloading region based on the grapple operating region and the difference region. Therefore, the iron scrap sorting device 100 can determine the final region of interest to be a rectangular area (unloading region B-box) that includes the unloading region determined along the boundary indicating the difference region. Therefore, since a segmented image is acquired for the area of ​​interest relative to the unloading area determined along the boundary indicating the difference area, and iron scrap classification information is obtained, there is an advantage in that the error rate due to changes in the surrounding area other than the unloading area can be reduced. In one embodiment, the iron scrap classification device 100 may determine the area of ​​interest based on the operating state of the grapple regarding whether the grapple contains at least one or more iron scraps when the grapple is included in any part of the vertical area of ​​the loading equipment for a predetermined time or longer. For example, the iron scrap classification device 100 can determine whether or not iron scrap unloading has occurred based on the time the grapple is included in any part of the vertical area of ​​the loading equipment. If the time the grapple is included in the part of the area is 1 hour or longer, the iron scrap classification device 100 can determine that unloading of at least one or more iron scraps has occurred. In one embodiment, the 1 hour may be a time preset to correspond to the case in which unloading of iron scraps occurs during the cargo handling process. In addition, the part of the area may include multiple areas that are updated as the grapple moves.Therefore, if the sum of the times in which the grapple is included in each of the multiple partial areas of the scrap iron sorting device 100 is 1 hour or longer, it can be determined that unloading of the scrap iron has occurred. The scrap iron sorting device 100 can also determine the partial area in which the grapple is included for the longest time as the unloading area. In other words, in one embodiment, 1 hour can be the time used as a criterion for determining whether or not unloading of the scrap iron has occurred. In the scrap iron sorting device 100 according to one embodiment, if the grapple is included in a partial area for 2 hours or longer, which is longer than 1 hour (for example, half the time of 1 hour), the operating state of the grapple can be obtained for each of the multiple loading state images captured during the loading process, indicating whether the grapple contains at least one piece of scrap iron. If the time in which the grapple is included in a partial area is 2 hours or longer, it may be determined that unloading has not occurred. For example, if the second time is longer than the first time by a predetermined amount, and the grapple is included in a portion of the area for a period of two hours or more, there is a high probability that the grapple is currently operating in real time. Therefore, the iron scrap sorting device 100 can acquire the grapple's operating state and determine the region of interest based on the operating state. For example, by acquiring the grapple's operating state, it is possible to identify a loading state image from among multiple loading state images in which the grapple contains at least one or more pieces of iron scrap. The iron scrap sorting device 100 can determine the boundary region in which the grapple was located when the grapple's operating state first contained iron scrap as the region of interest, or it can determine the boundary region in which the grapple was located when the grapple's operating state last contained iron scrap as the region of interest. If the time the grapple is included in a portion of the area is two hours or more, it may correspond to a situation where the grapple remains in the vertical region of the loading equipment for a long period of time.In other words, if there is a loading image in which the grapple contains at least one or more pieces of scrap iron, it can correspond to cases where the grapple changes the position of multiple pieces of scrap iron rather than unloading multiple pieces of scrap iron from the loading equipment. Therefore, the scrap iron sorting device 100 may determine the region of interest based on the initial and final points in time when the grapple contains the scrap iron. In the case of the region of interest corresponding to the initial point in time, it may be analyzed to resemble a state in which some of the scrap iron has been unloaded, and in the case of the region of interest corresponding to the final point in time, it may be analyzed to resemble a state in which some of the scrap iron has been loaded. Therefore, the scrap iron sorting device 100 can determine the region of interest for the area where the scrap iron is unloaded based on the case where the grapple remains in the vertical region of the loading equipment and then disappears, and if the grapple remains in the vertical region of the loading equipment for two hours or more, it can determine the region of interest for the area to which the scrap iron is moved based on the grapple's operating state. Therefore, it has the effect of being able to perform analysis and classification for each of multiple pieces of scrap iron that may be located according to various situations. In other embodiments, the scrap iron sorting device 100 can acquire an operation pattern based on the grapple operation state. For example, if the time the grapple is included in a certain area is two hours or more, the scrap iron sorting device 100 can analyze a plurality of continuously captured loading state images to determine whether the grapple operation state alternates between a state containing scrap iron and a state not containing scrap iron. If an operation pattern is acquired in which states containing scrap iron and states not containing scrap iron alternate, the scrap iron sorting device 100 may differently determine the position and number of regions of interest based on the sequential positions of the states containing scrap iron and the states not containing scrap iron as the number of repetitions and the passage of time progress.For example, if the iron scrap sorting device 100 has only one iteration, it has moved the positions of some of the iron scraps out of a group of iron scraps once, so the number of regions of interest can be determined to be two, and the positions of the regions of interest can be determined as the region corresponding to the position of the grapple when the grapple first contains the iron scrap and the region corresponding to the position of the grapple when the grapple last contains the iron scrap. In addition, the size of the regions of interest can be determined by the change between the previous loading state image (image before update) and the current loading state image (updated image). If the iron scrap sorting device 100 has only one iteration, the regions of interest may be updated based on the overlap probability for the two regions of interest. For example, if the sequential positions of the state containing iron scrap and the state not containing iron scrap are less than a predetermined distance (for example, less than 10 percent of the lateral length of the loading equipment), the iron scrap sorting device 100 may overlap the two areas of interest to update them into one area of ​​interest and perform a primary image analysis, perform a secondary image analysis on the two areas of interest, and then subdivide the two areas of interest to update them into three areas of interest and perform a tertiary image analysis. In other words, if the sequential positions of the state containing iron scrap and the state not containing iron scrap are less than a predetermined distance, the iron scrap sorting device 100 can provide iron scrap classification information through a tertiary image analysis process. Since the probability of overlap for areas of interest is high when the distance is less than a predetermined distance, the two areas of interest can be overlapped to determine a single area of ​​interest, and item information and grade information can be obtained for each of the target iron scraps, which are any one of the multiple iron scraps, based on the layer information. Furthermore, analysis can be performed on each of the two areas of interest to obtain item information and grade information for each of the target iron scraps. Furthermore, by performing an analysis on each of the three regions of interest, which are divided based on the distance between the midpoints of the grapples in the state containing iron scrap and the state without iron scrap, and a length that is one-third of the average of the previously set distances, item information and grade information for each of the target iron scraps can be obtained.Therefore, by comparing and analyzing the results of the tertiary image analysis process, the analytical accuracy for each target scrap iron can also be improved. In other embodiments, if the number of repetitions is two or more, the scrap iron sorting device 100 can determine that, as described above, it is not a situation in which the process of analyzing one area of ​​interest in a manner similar to that of unloaded scrap iron, and analyzing another area of ​​interest in a manner similar to that of loaded scrap iron, is not applicable. In other words, if the number of repetitions is two or more, it can be determined that this corresponds to the act of mixing multiple scrap iron pieces, and does not correspond to unloading and / or loading. Therefore, if the number of repetitions is two or more, the scrap iron sorting process may be terminated, and the process may be reset and re-executed at the point when the grapple is not included in some areas (when the grapple disappears).

[0062] Figure 9 is a diagram illustrating an example in which an iron scrap sorting device 100 according to one embodiment of the present disclosure acquires layer information for each layer change point by stereotyping.

[0063] Referring to Figure 9, the iron scrap sorting device 100 according to one embodiment can acquire a first layer change point when the height of multiple iron scraps changes to a critical length or more, based on depth information acquired by stereotyping. The iron scrap sorting device 100 can also acquire a second layer change point when the volume of multiple iron scraps changes to a critical percentage or more, based on the average volume of the loading equipment. Therefore, the iron scrap sorting device 100 can acquire layer information based on at least one of the first and second layer change points. In one embodiment, the iron scrap sorting device 100 can perform a layer change measurement process. A layer may be a concept of a unit corresponding to the layer of iron scrap that has changed after at least one iron scrap has been unloaded using a grapple. For example, when iron scrap is loaded so that it fills the loading equipment, it may be called Layer-3, and when iron scrap is loaded to exceed the height of the loading equipment, it may be called Layer-3.5. Furthermore, during the handling process, when the height of the iron scrap in the loading equipment changes beyond a critical length, the layer can be updated to an even smaller layer (e.g., Layer-2 or Layer-1) at any point. In one embodiment, when determining item and grade information for the target iron scrap using layer information determined based on depth information acquired by stereotypes, it has the advantage of being insensitive to changes due to movement, etc., and allowing for the application of a higher level of AI or machine learning due to the time between layers. As illustrated in Figure 9, the iron scrap sorting device 100 can determine the layer to Layer-3 when the loading equipment is full of multiple pieces of iron scrap, and can determine to an even lower layer as the loading height decreases. In Figure 9, it can be seen that Layer-1 has multiple pieces of iron scrap loaded at a low height and the volume of multiple pieces of iron scrap is small, while in Figure 9, Layer-3 has multiple pieces of iron scrap loaded at a high height and the volume of multiple pieces of iron scrap is large. The iron scrap sorting device 100 can acquire depth information by obtaining depth maps for each layer.In one embodiment, when the average unloading time is around 5 minutes (300 seconds), unloading is performed approximately 20 times using a grapple, and there are roughly 3 layers, determining item information and grade information on a grapple-by-grapple basis requires inference within 15 seconds (300 seconds / 20 times). On the other hand, determining item information and grade information on a layer-by-layer basis only requires determination within 100 seconds (300 seconds / 3 layers), which has the advantage of allowing the application of a higher level of AI or machine learning. In one embodiment, the iron scrap sorting device 100 can determine the point at which the height of multiple iron scraps falls below a critical length (approximately 0.4 m) as the first layer change point. The iron scrap sorting device 100 can also obtain average volume information for the loaded equipment from the administrator account. Based on the obtained average volume information, the iron scrap sorting device 100 can determine the point at which the volume of multiple iron scraps decreases to above a critical percentage (approximately 33 percent) relative to the average volume of the loaded equipment as the second layer change point. Therefore, the scrap iron sorting device 100 can determine the layer at whichever point in time is earlier, based on the first layer change time and / or the second layer change time, or at the average time of the first and second layer change times. When analyzing the divided images, the scrap iron sorting device 100 can further utilize the depth information obtained in at least one of the determined layers to obtain item information and grade information for the target scrap iron. That is, it can measure the changes between each image, which are updated layer by layer based on the layer change time. In other embodiments, the scrap iron sorting device 100 can assign different weights to the first and second layer change times. For example, since the importance of height (depth) may be higher than the importance of volume when utilizing layer information, the scrap iron sorting device 100 can assign weights of 7:3 to the first and second layer change times, respectively. Alternatively, the scrap iron sorting device 100 may determine the ratio differently depending on the initial loading state in which multiple pieces of scrap iron are loaded.For example, the importance of height (depth) can be determined differently depending on the initial loading state and the height at which it is loaded. For instance, if multiple pieces of scrap iron are loaded to correspond to the full height of the loading equipment based on the initial loading state, weight values ​​can be assigned at a ratio of 8:2 at the first layer change point and the second layer change point. If multiple pieces of scrap iron are loaded to correspond to 90% or more of the loading equipment height, weight values ​​can be assigned at a ratio of 7:3 at the first layer change point and the second layer change point. If multiple pieces of scrap iron are loaded to correspond to 80% or more but less than 90% of the loading equipment height, weight values ​​can be assigned at a ratio of 6:4 at the first layer change point and the second layer change point. If multiple pieces of scrap iron are loaded to correspond to less than 80% of the loading equipment height, weight values ​​can be assigned at a ratio of 8:2 and 5:5 at the first layer change point and the second layer change point. Therefore, by updating the ratio at which weight values ​​are assigned so that the importance of volume relative to length increases depending on the loading height of the loading equipment, the average time at which layers are acquired is determined, thereby improving efficiency in utilizing layer information. In other embodiments, the iron scrap sorting device 100 may determine the importance of volume differently depending on the size of the loading equipment. For example, if the total volume of the loading equipment is small, less than the previously set volume, the sensitivity of volume may be high, so weight values ​​may be assigned at a ratio of 3:7 at the first layer change time and the second layer change time. Also, if the total volume of the loading equipment is larger than a certain multiple (3 times) of the previously set volume, the critical length and critical percentage may be updated. For example, if the total volume of the loading equipment is larger than a certain multiple, the height may not decrease significantly even if a large amount of iron scrap is unloaded, so the critical length can be reduced to a length corresponding to 80% of the conventional critical length (e.g., about 0.32 m), and the critical percentage can be increased to a percentage corresponding to 120% of the conventional critical percentage (e.g., about 39.6 percent). Therefore, efficiency can be improved by adjusting the sensitivity of height and volume according to the situation. The respective numerical values ​​and percentages mentioned above may be changed flexibly depending on the situation and may be modified and set by the administrator.

[0064] Figure 10 is a schematic diagram illustrating an example in which a scrap iron sorting apparatus 100 according to one embodiment of the present disclosure performs semantic segmentation and instance segmentation.

[0065] Referring to Figure 10, the iron scrap sorting device 100 according to one embodiment can perform semantic segmentation, which divides objects of the same type into the same region, and / or instance segmentation, which divides objects of the same type into different regions, when dividing the types of objects in an image and their boundaries. That is, the AI ​​learning model can acquire the type and location of iron scrap from the loaded state image, and acquire item information and grade information corresponding to each piece of iron scrap. The iron scrap sorting device 100 according to one embodiment can perform semantic segmentation and instance segmentation in parallel. That is, instance segmentation may be performed after semantic segmentation, or semantic segmentation may be performed after instance segmentation. For example, as shown in Figure 10, the scrap iron sorting device 100 can perform semantic segmentation, dividing identical scrap iron into the same region. For each segmented image of a group of scrap iron objects obtained through semantic segmentation, instance segmentation is performed to divide it into individual, distinct scrap iron regions, thereby obtaining segmented images for single objects. As shown in Figure 10, the scrap iron sorting device 100 can perform semantic segmentation, labeling (dividing) multiple scrap iron pieces into group objects, and can perform instance segmentation, labeling (dividing) multiple scrap iron pieces into individual objects. Conversely, the scrap iron sorting device 100 may also perform semantic segmentation on regions from which the region corresponding to a single object has been excluded, after instance segmentation has been performed and image division for a single object has been completed. Therefore, even in regions where instance segmentation is not accurately performed, it is possible to obtain further segmentation images for the clustered objects, thus potentially improving accuracy. This can be explained with reference to Figure 11.

[0066] Figure 11 is a schematic diagram illustrating an example in which an iron scrap sorting apparatus 100 according to one embodiment of the present disclosure further acquires a segmented image for a group of objects in an area from which the area corresponding to a single object has been excluded.

[0067] Referring to Figure 11, the iron scrap sorting device 100 according to one embodiment can perform semantic segmentation on the region from which the region corresponding to a single object obtained by instance segmentation in Figure 10 has been excluded, thereby obtaining a segmented image for a group of objects. Therefore, after performing semantic segmentation, instance segmentation may be further performed on the region corresponding to the group of objects. Thus, as shown at the bottom of Figure 11, even in the region from which the region corresponding to a single object has been excluded, the regions of each of the multiple iron scraps can be further divided into single objects. Although Figure 11 describes an example of performing semantic segmentation and instance segmentation on multiple iron scraps in the entire loading equipment, it is not limited to this, and semantic segmentation and instance segmentation can be applied identically to the region of interest. Thus, the iron scrap sorting device 100 acquires segmented images for each of the multiple iron scrap regions by using semantic segmentation and instance segmentation in parallel, which has the effect of improving the accuracy of item information and grade information for each of the target iron scraps.

[0068] According to one embodiment, by performing image analysis and image classification using segmentation and classification models, it is possible to acquire segmented images for both single objects and crowd objects, thereby improving the accuracy of iron scrap classification. Furthermore, by acquiring layer information based on monotypes and stereotypes, the accuracy of image classification and analysis for unloading areas can be improved, and by determining areas of interest based on the grapple operation area, the accuracy of detecting unloading areas can be improved. In addition, by acquiring segmented images by using semantic segmentation and instance segmentation in parallel, it is possible to complement areas where each segmentation is not performed accurately.

[0069] Various embodiments of this disclosure may be embodied in software that includes one or more instructions stored in a storage medium (e.g., memory) readable by a machine (e.g., a display device or a computer). For example, the machine's processor (e.g., processor 220) may call and execute at least one of the one or more instructions stored in the storage medium. This allows the machine to be operated to perform at least one function by the one or more instructions called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, “non-transitory” simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily on the storage medium.

[0070] According to one embodiment, the methods relating to the various embodiments disclosed herein may be provided in a computer program product. The computer program product may be traded as a commodity between sellers and buyers. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or online (e.g., downloaded or uploaded) through an application store (e.g., PlayStore™), or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated on a device-readable storage medium such as the memory of a manufacturer's server, an application store server, or an intermediary server.

[0071] Although the present invention has been described with reference to the illustrated drawings, it is not limited by the disclosed embodiments and drawings, and a person with ordinary skill in the art relating to these embodiments will understand that it can be embodied in modified forms without departing from the essential characteristics described above. Therefore, the disclosed methods should be considered in an explanatory rather than restrictive manner. Even if the effects of the configuration of the present invention are not explicitly described when describing the embodiments, predictable effects may be recognized by the configuration in question. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within an equivalent scope should be interpreted as being included in the present invention. [Explanation of Symbols]

[0072] 100: Iron scrap sorting machine 110: Receiver 120: Processor

Claims

1. In a method for providing iron scrap classification information based on the handling process, The receiving unit acquires images of the loading condition of multiple pieces of scrap iron loaded on the loading equipment, which are captured during the loading and unloading process; The processor is updated as the handling process progresses, and acquires layer information determined by the heights of the multiple pieces of scrap iron; The processor acquires a certain region within the loading state image that includes the grapple boundary as the grapple operating region; The step in which the processor determines a portion of the region, including the grapple operation region, as a region of interest; The processor obtains a segmented image of a target scrap iron that is one of the plurality of scrap iron pieces and is included in the region of interest; The processor performs classification on the divided image and obtains item information, which is characteristic information including weight information and lightweight information of the target scrap iron, by utilizing depth information that allows confirmation of the height of each of the multiple scrap iron pieces included in the layer information; and A method comprising the step of providing iron scrap classification information including the item information of the processor.

2. The method for acquiring the aforementioned layer information is This includes monotypes using one camera or stereotypes using two or more cameras. The method according to claim 1, wherein the monotype utilizes depth information obtained through analysis of changes in the loading platform wall surface obtained from images acquired by the single camera.

3. The step of determining the area of ​​interest is The method according to claim 1, wherein the processor determines the region of interest based on a comparison result between a first loading state image corresponding to a first time point and a second loading state image corresponding to a second time point that occurs later in time than the first time point.

4. The step of determining the area of ​​interest is The method according to claim 3, wherein the processor determines the region of interest based on the difference region between the first loading state image and the second loading state image.

5. The step in which the processor acquires the segmented image involves using a segmentation model to acquire the segmented image including the target iron scrap from the loaded state image, The method according to claim 1, wherein the step of the processor acquiring the item information involves acquiring the item information corresponding to the target iron scrap using a classification model that performs analysis on an image-by-image basis.

6. The step of obtaining the aforementioned segmented image is The step of the processor performing instance segmentation on the loading state image to obtain the segmented image for a single object; and The method according to claim 5, further comprising the step of the processor performing semantic segmentation on the loading state image to obtain the segmented image for the crowd object.

7. The step of obtaining the segmented image for the aforementioned crowd is The method according to claim 6, wherein the method is performed on the region from which the region corresponding to the single object in the loading state image has been excluded.

8. The method according to claim 2, wherein the stereotype utilizes depth information obtained through analysis of angular changes obtained from images acquired from two or more cameras located on the same plane for the same region.

9. The second time point is determined based on whether or not the grapple is included in any part of the vertical region of the loading equipment after the first time point. The step of determining the area of ​​interest is The method according to claim 4, wherein if the grapple is contained in the portion of the region for a predetermined time or longer, the processor determines the region of interest based on the operating state of the grapple to determine whether the grapple contains at least one or more pieces of iron scrap.

10. The method for acquiring the aforementioned layer information is The processor obtains a first layer change point in time when the height of the multiple iron scraps changes to a critical length or more, based on the depth information obtained by the stereotype. The processor acquires a second layer change point in which the volume of the multiple pieces of scrap iron changes to a critical percentage or more, based on the average volume of the loaded equipment. The method according to claim 8, wherein the processor acquires the layer information based on at least one of the first layer change time and the second layer change time.

11. In a scrap iron sorting device that provides scrap iron sorting information based on the handling process, A receiving unit that acquires images of the loading state of multiple pieces of scrap iron loaded on the loading equipment during the loading and unloading process; and As the aforementioned cargo handling process progresses, layer information determined by the height of the multiple pieces of scrap iron is obtained, Within the aforementioned loading state image, a certain region containing the boundary of the grapple is acquired as the grapple operation region. A portion of the region including the aforementioned grapple operation area is determined to be the region of interest, A segmented image is obtained for the target scrap iron that is included in the aforementioned region of interest and is one of the multiple scrap iron pieces. Classification is performed on the segmented image, and using the depth information included in the layer information, which allows confirmation of the height of each of the multiple pieces of scrap iron, item information, which is characteristic information including weight information and lightweight information of the target scrap iron, is obtained. A scrap iron sorting apparatus comprising: a processor that provides scrap iron sorting information including the aforementioned item information;

12. The method for acquiring the aforementioned layer information is This includes monotypes using one camera or stereotypes using two or more cameras. The iron scrap sorting apparatus according to claim 11, wherein the monotype utilizes depth information obtained through analysis of changes in the loading platform wall surface obtained from the image acquired by the single camera.

13. The aforementioned processor The iron scrap sorting apparatus according to claim 11, wherein the region of interest is determined based on a comparison result between a first loading state image corresponding to a first time point in time and a second loading state image corresponding to a second time point that occurs later in time than the first time point.

14. The aforementioned processor The iron scrap sorting apparatus according to claim 13, wherein the region of interest is determined based on the region of difference between the first loading state image and the second loading state image.

15. A computer-readable recording medium that stores a program for causing a computer to perform the method described in any one of claims 1 to 10.

Images