Method and system for image-based recognition of foreign materials in recycled aggregates using deep learning

Abstract

A method for recognizing images of foreign substances in recycled aggregate according to an embodiment of the present invention comprises: a shooting area setting step of setting a shooting area on the upper part of a conveyor through which recycled aggregate is transported; an image acquisition step of capturing the recycled aggregate passing through the shooting area as a plurality of consecutive frame images; an object candidate extraction step of analyzing at least one image characteristic among color, shape, and texture for each shooting area to extract object candidates distinguishable from the recycled aggregate; an object position calculation step of converting the image coordinates of the object candidates into the actual transport coordinate system of the conveyor to calculate the conveyor width direction position and the transport direction position of the object candidates; an object characteristic calculation step of calculating object characteristic information including at least one of size information, shape information, color information, and texture information for each object candidate; an object feature extraction step of tokenizing the object characteristic information and, to ensure low latency, extracting information among the object characteristic information whose contribution to foreign substance determination is greater than or equal to a preset standard and transmitting it to a deep learning model; a foreign substance determination step of inputting the object characteristic information into the deep learning model to calculate the type of foreign substance of the object candidate; and the conveyor coordinates, type of foreign substance, and object characteristic information for each object candidate. It may include a foreign substance map generation step for generating a foreign substance map, a control information output step for outputting the foreign substance map to a blower control device, a data cleaning step including data collection, cleaning, preprocessing, and data transmission for the deep learning model training process, and a learning process automation step in which the deep learning model is retrained when a preset condition is met using the data cleaned in the data cleaning step.

Description

Technology Field

[0001] This invention is the result of a project carried out with support from the '2026 Eco-Startup Support Program,' promoted by the Ministry of Climate, Energy and Environment and operated by the Korea Environmental Industry & Technology Institute.

[0002] The present invention relates to the fields of computer vision and construction waste treatment technology.

[0003] More specifically, the invention relates to a method for recognizing foreign substances in recycled aggregate images, wherein a shooting area is set for a plurality of consecutive frame images of recycled aggregate being transported on a conveyor, a foreign substance map including foreign substance type and object characteristic information is generated, and the generated foreign substance map is output to a blower control device. Background Technology

[0004] As the volume of construction waste increases annually, the importance of technologies to recycle it into recycled aggregate is growing. Recycled aggregate generated during the intermediate treatment of construction waste contains various lightweight foreign materials, such as plastic, wood, vinyl, paper, rubber, and glass fragments, in addition to concrete fragments and soil. Therefore, a process to effectively separate and remove these foreign materials is essential to meet quality standards for recycled aggregate.

[0005] Conventionally, physical separation methods such as magnetic separation and vibrating sieve separation have been primarily used. Magnetic separation is a method of separating metallic foreign substances using magnets, and sieve separation is a method of separating aggregates based on particle size. However, these physical separation methods had limitations in that it was difficult to secure sufficient separation performance for fine foreign substances with a specific gravity or particle size similar to that of aggregates, or with a particle diameter of 25 mm or less.

[0006] To overcome these limitations, attempts are being made to detect foreign substances contained in recycled aggregates via images and to remove them using blower technology, computer vision, and artificial intelligence technologies.

[0007] Since a significant portion of the foreign substances contained in construction waste consists of lightweight materials such as wood and vinyl that have low specific gravity and are easily dispersed by airflow, a sorting method utilizing the difference in specific gravity between foreign substances and recycled aggregates is effective. In particular, among sorting methods utilizing density differences, the method using blowing air is economically superior compared to wet sorting methods that use separate media such as water, as it involves lower equipment and operating costs and is easy to apply to continuous processes. Meanwhile, metallic foreign substances that are difficult to remove by blowing air can be largely eliminated through a magnetic separation process. AI-based image analysis technology can automatically detect and classify objects within images, and can be utilized to recognize fine foreign substances that are difficult to remove using conventional physical sorting methods.

[0008] However, at recycled aggregate sorting sites, image quality is inconsistent due to dust, changes in illumination, shadows, overlap between aggregates, and irregular surface shapes. In particular, there is a problem in that accurate detection of fine foreign substances smaller than 25mm is difficult because they are small in size and have an appearance similar to surrounding aggregates. Furthermore, in order to detect foreign substances in real-time on recycled aggregates moving on a conveyor, it is necessary to simultaneously ensure detection accuracy and processing speed, and there is a continuous demand for improvement in this regard. The problem to be solved

[0009] The present invention aims to provide a method for recognizing images of foreign substances in recycled aggregate, wherein recycled aggregate being transported on a conveyor is captured as a plurality of consecutive frame images, a shooting area is set, at least one image characteristic among color, shape, and texture is analyzed for each shooting area to extract object candidates that are distinguishable from the recycled aggregate, and the image coordinates of the object candidates are converted into the actual transport coordinate system of the conveyor to calculate conveyor coordinates for each object candidate.

[0010] In addition, the present invention aims to provide object-unit control linkage data so that a downstream airflow control device can determine airflow conditions such as airflow position, airflow timing, and airflow intensity using the foreign matter map by calculating object characteristic information including at least one of size information, shape information, color information, and texture information for each object candidate, inputting the object characteristic information into a deep learning model to calculate the type of foreign matter of the object candidate, generating a foreign matter map including conveyor coordinates, the type of foreign matter, and object characteristic information for each object candidate, and outputting the generated foreign matter map to an airflow control device.

[0011] In addition, the present invention aims to minimize false detection that may occur due to dust, shadows, lighting reflections, or camera shake by tracking the positional change of the same object candidate in a plurality of consecutive frame images, determining whether the positional change matches a reference movement amount corresponding to the conveyor's transport speed, and determining object candidates that deviate from the allowable range of the reference movement amount as noise objects.

[0012] Furthermore, the present invention aims to enable differential airflow control according to the type and characteristics of foreign substances of an object candidate by calculating a scattering difficulty, which indicates the degree to which an object candidate is difficult to scatter by airflow, using at least one of the type of foreign substance, location information, size information, shape information, and detection reliability of the object candidate, and including this in a foreign substance map and outputting it to a blower control device.

[0013] In addition, the present invention aims to enable stable extraction of object candidates even in obscured and overlapping environments at recycled aggregate sorting sites by determining whether an object candidate is a partially exposed object partially obscured by surrounding aggregates or an overlapping object overlapping with surrounding aggregates based on at least one of the boundary integrity of the object candidate, the degree of contact with surrounding aggregates, and the exposed area of ​​the object candidate. means of solving the problem

[0014] A method for recognizing images of foreign substances in recycled aggregate according to an embodiment of the present invention comprises: a shooting area setting step of setting a shooting area on the upper part of a conveyor through which recycled aggregate is transported; an image acquisition step of capturing the recycled aggregate passing through the shooting area as a plurality of consecutive frame images; an object candidate extraction step of analyzing at least one image characteristic among color, shape, and texture for each shooting area to extract object candidates distinguishable from the recycled aggregate; an object position calculation step of converting the image coordinates of the object candidates into the actual transport coordinate system of the conveyor to calculate the conveyor width direction position and the transport direction position of the object candidates; an object characteristic calculation step of calculating object characteristic information including at least one of size information, shape information, color information, and texture information for each object candidate; an object feature extraction step of tokenizing the object characteristic information and, to ensure low latency, extracting information among the object characteristic information whose contribution to foreign substance determination is greater than or equal to a preset standard and transmitting it to a deep learning model; a foreign substance determination step of inputting the object characteristic information into the deep learning model to calculate the type of foreign substance of the object candidate; and the conveyor coordinates, type of foreign substance, and object characteristic information for each object candidate. It may include a foreign substance map generation step for generating a foreign substance map, a control information output step for outputting the foreign substance map to a blower control device, a data cleaning step including data collection, cleaning, preprocessing, and data transmission for the deep learning model training process, and a learning process automation step in which the deep learning model is retrained when a preset condition is met using the data cleaned in the data cleaning step.

[0015] After the above image acquisition step, the method may further include an image preprocessing step of performing at least one of illumination correction, shadow removal, shake correction, background removal, and conveyor belt area masking on the plurality of consecutive frame images.

[0016] After the image acquisition step, the method may further include a multi-frame tracking step of tracking the position change of the same object candidate in the plurality of consecutive frame images and determining whether the position change matches a reference movement amount corresponding to the conveyor's transport speed.

[0017] In the multi-frame tracking step above, if the position change of the same object candidate deviates from the allowable range of the reference movement amount, a noise correction step for determining the object candidate as a noise object may be further included.

[0018] The above foreign substance determination step calculates the detection reliability for the type of foreign substance along with the type of foreign substance of the object candidate using the deep learning model, and the above foreign substance map generation step can generate a foreign substance map that further includes the detection reliability for each object candidate.

[0019] After the foreign substance determination step, the method further includes a scattering difficulty calculation step that calculates a scattering difficulty indicating the degree to which the object candidate is difficult to scatter by airflow using at least one of the foreign substance type, location information, size information, shape information, and detection reliability of the object candidate, and the foreign substance map generation step may generate a foreign substance map that further includes the scattering difficulty for each object candidate.

[0020] The above-mentioned step for calculating the difficulty of scattering can calculate the difficulty of scattering by determining a preset reference scattering value for each type of foreign substance and applying a correction factor based on at least one of the area, major axis, minor axis, contour complexity, circularity, and aggregate stacking degree of the object candidate to the reference scattering value.

[0021] After the object candidate extraction step, the method may further include a step of determining whether the object candidate is a partially exposed object partially obscured by surrounding aggregate or an overlapping object overlapping with surrounding aggregate, based on at least one of the boundary completeness of the object candidate, the degree of contact with surrounding aggregate, and the exposure area of ​​the object candidate. Effects of the invention

[0022] The present invention can provide object-unit control linkage data that enables a downstream airflow control device to precisely determine airflow conditions, such as airflow position, airflow timing, and airflow intensity, on an object unit basis by generating a foreign substance map including conveyor coordinates, foreign substance type, and object characteristic information for each object candidate and outputting it to an airflow control device.

[0023] In addition, the present invention determines noise objects by determining whether the position change of the same object candidate in a plurality of consecutive frame images matches a reference movement amount corresponding to the conveyor transport speed, thereby minimizing false detection caused by dust, shadows, lighting reflections, or camera shake and preventing malfunction of the blower control device.

[0024] In addition, the present invention calculates the difficulty of scattering according to the type and characteristics of foreign substances of object candidates and includes it in a foreign substance map, thereby enabling a downstream blower control device to determine differential blower conditions based on the area, major axis, minor axis, contour complexity, circularity, and degree of aggregate stacking, even for foreign substances of the same type.

[0025] In addition, the present invention enables stable extraction of object candidates even in actual sorting sites where recycled aggregates are stacked or transported overlappingly on a conveyor by determining whether an object candidate is a partially exposed object partially obscured by surrounding aggregates or an overlapping object.

[0026] In addition, the present invention can secure high detection accuracy even for fine foreign substances with small particle sizes by automatically distinguishing and classifying recycled aggregates and various types of foreign substances using a deep learning model. Brief explanation of the drawing

[0027] FIG. 1 is a block diagram of the overall configuration of a recycled aggregate foreign substance image recognition system according to one embodiment of the present invention. FIG. 2 is a flowchart showing the processing flow of a method for recognizing images of foreign substances in recycled aggregate according to one embodiment of the present invention. FIG. 3 is a diagram illustrating a multi-frame tracking and noise object determination process according to an embodiment of the present invention. Specific details for implementing the invention

[0028] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. Prior to this, terms and words used in this specification and claims should not be interpreted as being limited to their ordinary or dictionary meanings, and should be interpreted in a meaning and concept consistent with the technical spirit of the present invention, based on the principle that the inventor can appropriately define the concept of the terms to best describe his invention.

[0029] Therefore, the embodiments described in this specification and the configurations illustrated in the drawings are merely the most preferred embodiments of the present invention and do not represent all of the technical ideas of the present invention; thus, it should be understood that various equivalents and modifications that can replace them may exist at the time of filing this application.

[0030] In this specification, "conveyor width direction" means a direction perpendicular to the conveying direction of the conveyor (100), and "conveyor conveying direction" means the direction in which the recycled aggregate on the conveyor (100) is conveyed.

[0031] In addition, in this specification, "object candidate" refers to an area extracted and distinguished from recycled aggregate as a result of image characteristic analysis for each shooting area (A), and "foreign substance" may refer to a type other than recycled aggregate, such as plastic, wood, vinyl, paper, rubber, or glass fragments, among the object candidates, determined by a deep learning model.

[0032] In addition, in this specification, "deep learning model" may refer to an artificial intelligence model that learns the image characteristics of recycled aggregates and foreign substances using a plurality of training data. According to one embodiment, the deep learning model may be trained to perform object detection, object classification, image segmentation, or a combination thereof, and may include, but is not limited to, a Convolutional Neural Network (CNN), You Only Look Once (YOLO), Faster R-CNN, EfficientNet, Vision Transformer (ViT), or a neural network structure equivalent thereto.

[0034] FIG. 1 is a block diagram of the overall configuration of a recycled aggregate foreign substance image recognition system according to one embodiment of the present invention.

[0035] Referring to FIG. 1, a recycled aggregate foreign substance image recognition system according to one embodiment may include a conveyor (100), a camera (200), a lighting unit (300), an image processing unit (400), a foreign substance map generation unit (500), and a blower control device (600).

[0036] The conveyor (100) may be configured to transport recycled aggregate, from which particles larger than 25mm have been separated through primary sieving, to a blowing treatment line at the rear end via the shooting area (A) of the camera (200). According to one embodiment, the system of the present invention may include a 25mm sieve device at the front end of the conveyor (100), and the 25mm sieve device may primarily separate fine powder and particles larger than 25mm from the recycled aggregate generated in the construction waste treatment process, thereby allowing only recycled aggregate with a particle size of 25mm or less, which is the target of foreign substance recognition of the present invention, to be supplied to the conveyor (100).

[0037] However, the particle size range of the recycled aggregate to which the present invention is applied is not limited to 25 mm or less. Depending on the target production size of the recycled aggregate, the applied processing process, and the performance of the downstream blower, it may be applied to various particle size ranges ranging from fine aggregate of 5 mm or less to 25 mm or less, greater than 25 mm and less than 40 mm, greater than 40 mm and less than 75 mm, or greater than 75 mm and less than 150 mm. Accordingly, the blower control device can adjust at least one of the blower position, blower timing, and blower intensity in response to the target production size of the recycled aggregate.

[0038] A vibrating feeder may be further coupled to the conveyor (100). A vibrating feeder according to one embodiment may be placed at the top of the conveyor (100) or at the front of the conveyor (100) to allow the recycled aggregate to pass through the shooting area (A) in a single layer, thereby improving the accuracy of the camera (200)'s shooting and subsequent analysis.

[0039] When recycled aggregate is spread out in a single layer by a vibrating feeder, the overlap between aggregates is reduced, which can improve the accuracy of image characteristic analysis and object candidate extraction for each shooting area (A). In one embodiment, the vibrating feeder may be configured to secure a shooting section of size 1m x 1m, but the size of the vibrating feeder and the shooting section may be changed according to the width and processing speed of the conveyor (100).

[0040] For convenience of explanation in this specification, the shooting section is exemplarily described as having a size of 1m x 1m, but the shooting section of the conveyor (100) can be varied depending on the width of the conveyor (100) used in the construction site, and the width of the shooting section is not limited to a size of 1m x 1m.

[0041] According to one embodiment, an alignment feeder may be further provided at the front end of the conveyor (100) (not shown). The alignment feeder may include a plurality of guide ribs or guide grooves formed on its surface and may guide recycled aggregates and foreign substances to be arranged in a certain direction along the conveying direction of the conveyor (100). For example, vinyl pieces, wood pieces, fibrous foreign substances, or other foreign substances with a long axis shape may be oriented so that their long axis direction is substantially parallel to the conveying direction of the conveyor (100) as they pass through the alignment feeder. Accordingly, shape information and contour information of object candidates can be acquired more stably, and deviations in image characteristics for foreign substances of the same type can be reduced, thereby improving the detection accuracy and classification accuracy of the deep learning model.

[0042] The camera (200) can generate a plurality of continuous frame images by capturing a shooting area (A) set on the upper part of the conveyor (100). The camera (200) according to one embodiment may be a high-resolution industrial camera having a resolution of 4K or higher, and may be configured to precisely capture fine foreign substances with a particle size of 25mm or less.

[0043] The camera (200) can be controlled based on a single MCU (MicroControllable Unit) and can output a plurality of continuous frame images by continuously shooting a shooting area (A) at a predetermined frame rate. According to one embodiment, the camera (200) may further include a correction function for lighting conditions and can output images of a certain quality even under various on-site lighting conditions through functions such as automatic exposure, automatic white balance, and automatic focus.

[0044] The lighting unit (300) can be configured to provide constant lighting to the shooting area (A) to minimize dependence on natural light and maintain a constant shooting environment. In one embodiment, the lighting unit (300) can be implemented with an LED lighting module to ensure that consistent image characteristic analysis results are obtained regardless of the time of day or external lighting conditions. According to one embodiment, the lighting unit (300) can be implemented in various forms such as a white light LED, a multi-channel LED, or a polarizing LED, and a plurality of LED modules can be arranged to provide a uniform amount of light along the width direction of the conveyor (100).

[0045] The image processing unit (400) may be configured to extract object candidates that are distinguishable from recycled aggregate by analyzing at least one image characteristic among color, shape, and texture for each shooting area (A) for a plurality of consecutive frame images received from the camera (200).

[0046] The foreign substance map generation unit (500) may be configured to receive the conveyor coordinates and object characteristic information of the object candidate calculated from the image processing unit (400), calculate the type of foreign substance of the object candidate using a deep learning model, and generate a foreign substance map including the conveyor coordinates, type of foreign substance, and object characteristic information for each object candidate. The foreign substance map generation unit (500) may output the generated foreign substance map to the blower control device (600).

[0047] The blower control device (600) may be configured to determine blower conditions, such as the blower position, blower timing, and blower intensity of a blower device (not shown), using a foreign matter map received from a foreign matter map generation unit (500).

[0048] The image processing unit (400) and the foreign substance map generation unit (500) can be implemented as a control computer based on a single MCU (Microcontrollable Unit) and can be configured to perform real-time inference of a deep learning model to complete the determination of the type of foreign substance within a predetermined time on a cell-by-cell basis.

[0049] In one embodiment, the cell unit judgment time may be within approximately 2.8 seconds, but it is not limited to a delay time within the range that guarantees real-time performance. In this case, the blower control device (600) can calculate the expected position of the object candidate based on the conveying speed and elapsed time of the conveyor (100) until the object candidate judgment is completed, and determine the blower position and blower timing corresponding to the calculated expected position. Accordingly, even if a certain processing delay occurs in the judgment of the object candidate, real-time sorting processing is possible by performing blower at the time when the object candidate reaches the rear blower position.

[0050] In addition, in some embodiments of the present invention, the conveying speed of the conveyor (100), the vibration conditions of the vibrating feeder, and the opening and closing operation of the rear separator can be integratedly controlled by a PLC (Programmable Logic Controller) system, and the PLC system can be configured to variably control the conveying speed of the conveyor (100) or adjust the vibration intensity of the vibrating feeder according to the processing results from the image processing unit (400) and the foreign substance map generation unit (500).

[0052] FIG. 2 is a flowchart showing the processing flow of a method for recognizing images of foreign substances in recycled aggregate according to one embodiment of the present invention.

[0053] Referring to FIG. 2, a processing flow of a method for recognizing images of foreign substances in recycled aggregate according to one embodiment is illustrated. The method of FIG. 2 can be performed by a system described with reference to FIG. 1, and the order of each step may be changed or some steps may be performed simultaneously.

[0054] In the shooting area setting step (S10), the image processing unit (400) may set a shooting area (A) on the upper part of the conveyor (100). The shooting area (A) is an area where the camera (200) photographs recycled aggregate and foreign substances, and can be set so as to reliably detect foreign substances contained in the recycled aggregate.

[0055] In the image acquisition step (S20), the image processing unit (400) can capture a circulating aggregate passing through a shooting area (A) through a camera (200) as a plurality of consecutive frame images. According to one embodiment, the plurality of consecutive frame images can be acquired at a frame rate synchronized with the conveying speed of the conveyor (100), and thereby any object candidate on the conveyor (100) can be captured across the plurality of consecutive frame images.

[0056] According to one embodiment, the image processing unit (400) may further perform an image preprocessing step after the image acquisition step (S20), which involves performing at least one of illumination correction, shadow removal, shake correction, background removal, and conveyor belt area masking on a plurality of consecutive frame images. Illumination correction may be performed using techniques such as histogram equalization, adaptive contrast correction, or gamma correction, and shadow removal may be performed using brightness channel analysis or color channel-based shadow area identification.

[0057] Shake correction can be performed through an image stabilization technique based on matching image feature points between consecutive frames, and background removal can be performed by masking the conveyor belt area by pre-learning the color or texture information of the conveyor belt. The image preprocessing step serves to improve the accuracy of object candidate extraction in subsequent steps by correcting for changes in illumination, shadows, camera shake, background interference, and misrecognition of the conveyor belt area that may occur in actual sorting sites.

[0058] According to one embodiment, the image preprocessing step may be performed as a process to improve the quality of input data for a deep learning model, and the preprocessed image may be used to improve object detection accuracy and foreign substance classification accuracy.

[0060] In the object candidate extraction step (S30), the image processing unit (400) can extract object candidates that are distinguishable from recycled aggregate by analyzing at least one image characteristic among color, shape, and texture for each shooting area (A).

[0061] According to one embodiment, color characteristics may include channel-specific statistics (mean, standard deviation, etc.) in an RGB color space, an HSV color space, or a LAB color space, shape characteristics may include edge distribution, shape descriptors, or contour information within a shooting area (A), and texture characteristics may include texture descriptors extracted through a Gray Level Co-occurrence Matrix (GLCM) or Local Binary Pattern (LBP). However, such characteristic analysis techniques are merely exemplary and the scope of the present invention is not limited thereto.

[0062] According to one embodiment, after the object candidate extraction step (S30), the image processing unit (400) may further perform a step of determining whether the object candidate is a partially exposed object partially obscured by surrounding aggregate or an overlapping object overlapping with surrounding aggregate, based on at least one of the boundary completeness of the object candidate, the degree of contact with surrounding aggregate, and the exposure area of ​​the object candidate.

[0063] In the object position calculation step (S40), the image processing unit (400) can convert the image coordinates of the object candidate into the actual transport coordinate system of the conveyor (100) to calculate the conveyor width direction position and transport direction position of the object candidate.

[0064] According to one embodiment, the center point, centroid, or representative point of the bounding box of the object candidate may be used, and the position on the image may be transformed to correspond to the actual conveyor position using preset position correction information or coordinate transformation information. This position calculation process may be performed using reference point-based calibration, a coordinate transformation matrix, or a homography transformation, but is not limited thereto.

[0065] The calculated conveyor width direction position and conveying direction position are coordinate components that constitute the "conveyor coordinates" of the foreign substance map described below. That is, the "conveyor coordinates" included in the foreign substance map in this specification can be understood to include both the conveyor width direction position and the conveyor conveying direction position calculated in the object position calculation step (S40). According to one embodiment, the conveyor coordinates may be expressed as two-dimensional coordinate values ​​with a predetermined reference point on the conveyor (100) as the origin, and various units such as millimeters, centimeters, or pixels may be used.

[0066] In the object characteristic calculation step (S50), the image processing unit (400) may calculate object characteristic information for each object candidate, including at least one of size information, shape information, color information, and texture information. In one embodiment, the size information may include the area, major axis, or minor axis of the object candidate, and the shape information may include the contour complexity or circularity of the object candidate. The area may be a value converted into the number of pixels corresponding to the object candidate or the actual area of ​​the conveyor, and the major axis and minor axis may be defined as the lengths of the long side and the short side of the minimum circumscribed rectangle surrounding the boundary of the object candidate. Contour complexity is an indicator representing the degree of curvature of the object candidate boundary, and may be defined as a value obtained by dividing the square of the object candidate's perimeter by the area, etc., and circularity is an indicator representing how close the object candidate is to a circle, and It can be defined as, etc.

[0067] Color information and texture information may be the result of integrating the results calculated from the image characteristic analysis for each shooting area (A) into object candidate units. According to one embodiment, color information may include the average RGB value of the object candidate area, a major color histogram or color moment, etc., and texture information may include texture indicators such as GLCM-based contrast, correlation, uniformity or entropy calculated from the object candidate area.

[0069] In the object feature extraction step (S60), the image processing unit (400) can tokenize the object feature information calculated in the object feature calculation step (S50) into a form that can be received by a deep learning model. In one embodiment, the tokenization may include a process of converting object feature information, such as size information, shape information, color information, and texture information, into a numeric vector or a feature token of a defined format.

[0070] In addition, the image processing unit (400) can select and extract only key information that contributes significantly to the determination of foreign substances among object characteristic information to ensure low latency, and transmit it to a deep learning model. Through this, the amount of transmitted data and computational load can be reduced, thereby securing real-time processing performance.

[0072] In the foreign substance determination step (S70), the image processing unit (400) can input image data and object characteristic information of the object candidate into a deep learning model to calculate the type of foreign substance of the object candidate. The deep learning model may be a model that has been trained in advance on image datasets of various types of foreign substances, such as plastic, wood, vinyl, paper, rubber, and glass fragments.

[0073] In one embodiment, the deep learning model may be implemented as a Convolutional Neural Network (CNN) based on frameworks such as TensorFlow or PyTorch, and at least one of various neural network structures such as ResNet, VGGNet, MobileNet, EfficientNet, YOLO, or Faster R-CNN may be employed, but is not limited thereto. In addition, the deep learning model may be implemented as an end-to-end classification model that receives not only object feature information but also the image region itself corresponding to the object candidate as input.

[0074] According to one embodiment, in the foreign substance determination step (S70), a deep learning model can be used to calculate the type of foreign substance of an object candidate along with the detection reliability for the said type of foreign substance. The detection reliability can be defined as a softmax probability value at the output of the deep learning model, and can be expressed as a normalized value between 0 and 1. A higher detection reliability may indicate a higher certainty of the classification result for the said type of foreign substance.

[0075] The types of foreign substances can be expanded by adding data and corresponding annotations during the learning phase, and the foreign substance classification process can be achieved by calculating the detection reliability for all registered foreign substances through this process.

[0076] In the actual use phase, for foreign substances exceeding the critical reliability, the type of foreign substance with the highest detection reliability can be used as a standard to determine subsequent treatment methods, such as blower intensity.

[0077] In addition, according to one embodiment, the deep learning model may be configured as a continuous learning algorithm that is continuously additionally learned by utilizing image data and object characteristic information accumulated during operation as training data, and thereby can continuously improve sorting accuracy by adapting to newly occurring foreign substance shapes or changes in the field environment.

[0078] According to one embodiment, a deep learning model may be pre-trained using training data for recycled aggregates, plastics, wood, vinyl, paper, rubber, and glass fragments. The training data may include images acquired at an actual sorting site, labeled object images, or augmented training images.

[0079] The deep learning model receives image data and object characteristic information of object candidates as input, determines whether the object candidate is recycled aggregate or foreign material, and if it is determined to be foreign material, calculates the type of foreign material of the object candidate. In addition, the deep learning model can calculate a detection confidence score along with the determination result.

[0080] According to one embodiment, when the same object candidate is detected in a plurality of consecutive frame images, the deep learning model or image processing unit (400) can determine the final type of foreign substance by combining the judgment results for the plurality of frames. Accordingly, misjudgment caused by temporary lighting changes, partial occlusion, or noise can be reduced.

[0082] In the foreign substance map generation step (S80), the foreign substance map generation unit (500) can generate a foreign substance map including conveyor coordinates, foreign substance type, and object characteristic information for each object candidate. In an embodiment where detection reliability is calculated together, the foreign substance map generation unit (500) can generate a foreign substance map that further includes detection reliability for each object candidate.

[0083] In addition, according to one embodiment, after the foreign substance determination step (S70), the foreign substance map generation unit (500) may further perform a scattering difficulty calculation step, which calculates a scattering difficulty indicating the degree to which an object candidate is difficult to scatter by airflow using at least one of the foreign substance type, location information, size information, shape information, and detection reliability of the object candidate, and the foreign substance map may further include a scattering difficulty for each object candidate.

[0084] According to one embodiment, the foreign substance map may be configured in the form of structured data including fields such as object ID, shooting frame number, shooting time, conveyor width direction coordinates, conveyor transport direction coordinates, foreign substance type, object area, major axis, minor axis, shape information, color information, texture information, detection reliability, and scattering difficulty.

[0085] In one embodiment, the foreign matter map may be serialized in a format such as JSON, XML, or CSV and transmitted to a blower control device (600), or transmitted via shared memory, a message queue, or an industrial communication protocol (e.g., Modbus, EtherCAT, etc.).

[0086] In the control information output step (S90), the foreign matter map generation unit (500) can output the generated foreign matter map to the blower control device (600). The blower control device (600) can determine blower conditions such as the blower position, blower timing, and blower intensity using the received foreign matter map.

[0088] In the data cleaning step (S100), data collection, cleaning, preprocessing, and data transmission for the training process of a deep learning model may be performed. In one embodiment, image data and object characteristic information acquired during operation may be collected as training data, annotations corresponding to the type of foreign substance may be assigned to each data, and a cleaning process that removes anotations, duplicates, and outlier data, and a preprocessing process including normalization or augmentation may be performed, and the cleaned training data may be transmitted to a training device or storage unit.

[0090] In the learning process automation step (S110), the learning process for continuous performance improvement of the deep learning model can be automated. In one embodiment, the deep learning model is automatically retrained periodically or when a preset condition is met using the training data refined in the data refinement step (S100), and the performance of the retrained model is evaluated, and if it satisfies the criteria, it can be updated and distributed as an operational model. Through this, the sorting accuracy can be continuously improved by adapting to newly occurring forms of foreign substances or changes in the field environment.

[0092] FIG. 3 is a diagram illustrating a multi-frame tracking and noise object determination process according to an embodiment of the present invention.

[0093] Referring to FIG. 3, a multi-frame tracking and noise object determination process according to one embodiment is illustrated. Referring to FIG. 3, the image processing unit (400) can track the position change of the same object candidate (O1) in the first frame (F1), the second frame (F2), and the third frame (F3). At this time, a reference movement amount (D) can be calculated based on the conveying speed of the conveyor (100) and the time interval between each frame. According to one embodiment, the reference movement amount (D) can be calculated according to the following formula.

[0094] Reference amount of movement (D) = Conveyor transfer speed × Time interval between frames

[0095] For example, if the conveying speed of the conveyor (100) is 0.5 m / s and the frame rate of the camera (200) is 30 fps, the time interval between frames is approximately 33.3 ms, and the reference movement amount (D) can be calculated as approximately 16.6 mm.

[0096] If the position change between frames of the same object candidate (O1) is within the allowable range of the reference movement amount (D), the image processing unit (400) can maintain the object candidate (O1) as a valid object candidate.

[0097] On the other hand, if the position change of an object candidate between frames exceeds the allowable range of the reference movement amount (D), the image processing unit (400) may determine the object candidate as a noise object. The allowable range may be defined as a predetermined ± ratio (e.g., ± 20%) or absolute value range centered on the reference movement amount (D), and may be adjusted according to the variability of the conveying speed of the conveyor (100) or the frame synchronization accuracy of the camera (200).

[0098] Accordingly, the likelihood of object candidates caused by dust, shadows, lighting reflections, camera shake, or temporary image noise being mistaken for actual foreign objects can be reduced. For example, a fake object candidate temporarily detected by shaking of the camera (200) may not be re-detected in subsequent frames at the same location or at a location corresponding to the conveyor transport speed, or may be judged as a noise object because the position change occurs in a direction unrelated to the conveyor transport direction, thus exceeding the allowable range of the reference movement amount (D).

[0099] The multi-frame tracking and noise correction described in relation to Fig. 3 support the fact that the present invention is not a simple image classification technology based on a single frame, but a real-time image recognition method that distinguishes between actual object candidates and noise objects using the conveying speed of the conveyor (100) and the time interval between frames.

[0100] To explain more specifically regarding the calculation of scattering difficulty according to one embodiment, the foreign substance map generation unit (500) determines a preset reference scattering value for each type of foreign substance and can calculate scattering difficulty by applying a correction coefficient based on at least one of the area, major axis, minor axis, contour complexity, circularity, and aggregate stacking degree of an object candidate to the reference scattering value. In one embodiment, the scattering difficulty can be calculated in the following form.

[0101] Flying Difficulty = Standard Flying Value Х (Correction factor)

[0102] Here represents the product of each correction factor, and the correction factor may represent a correction value corresponding to factors such as the area, major axis, minor axis, contour complexity, circularity, and aggregate stacking degree of the object candidate.

[0103] For example, the standard scatter value for vinyl-type foreign substances can be set to a relatively low value reflecting the characteristics of vinyl, which is light and has a large surface area, and the standard scatter value for wood-type foreign substances can be set to a relatively high value reflecting the characteristics of wood, which has a higher specific gravity than vinyl, and the standard scatter value for glass fragment-type foreign substances can be set to an even higher value reflecting the characteristics of glass, which has a very high specific gravity.

[0104] Furthermore, even for foreign materials of the same type of vinyl, vinyl fragments with a large surface area and flat surface are more easily dispersed by airflow than small vinyl fragments of the same type, which may result in a relatively low dispersion difficulty; conversely, foreign materials deeply deposited in aggregates are more difficult to disperse than foreign materials of the same type exposed on the surface, which may result in a relatively high dispersion difficulty. Additionally, foreign materials with complex contours or shapes close to circular may have less lift from airflow than flat foreign materials, which may result in a relatively high dispersion difficulty.

[0105] The scattering difficulty calculated in this way can be included in the foreign matter map and output to the blower control device (600), and the blower control device (600) can use this to determine differential blower conditions for each object candidate.

[0106] To explain more specifically regarding the determination of obscuration and overlap of an object candidate according to one embodiment, the image processing unit (400) can determine whether the object candidate is a partially exposed object partially obscured by surrounding aggregate or an overlapping object overlapped with surrounding aggregate based on at least one of the boundary completeness of the object candidate, the degree of contact with surrounding aggregate, and the exposure area of ​​the object candidate after the object candidate extraction step (S30).

[0107] Boundary completeness can be quantified based on whether the boundary of the object candidate forms a closed curve or whether the boundary is clearly identifiable, the degree of contact with surrounding aggregate can be quantified as the ratio of the portion of the object candidate's boundary that contacts the surrounding aggregate, and the exposed area can be quantified as the ratio of the visible area of ​​the object candidate to the estimated total area of ​​the object candidate. When the object candidate is determined to be a partially exposed object or an overlapping object, the image processing unit (400) calculates the type of foreign matter and object characteristic information of the object candidate using the image characteristics of the exposed area, and can also calculate the degree of occlusion or overlap as part of the object characteristic information.

[0108] The results of such occlusion and overlap judgment can be included as additional fields in the foreign matter map and output to the blower control device (600), and the blower control device (600) can perform responses such as increasing the blower intensity or adjusting the blower timing for object candidates with a high degree of occlusion or overlap.

[0109] The method for recognizing images of foreign substances in recycled aggregate according to the present invention can be performed continuously for 24 hours by automated operation of a camera (200), a lighting unit (300), an image processing unit (400), and a foreign substance map generation unit (500), and can maintain consistent sorting performance without separate manual sorting by an operator.

[0110] In addition, the foreign substance map generated by the foreign substance map generation unit (500) can be accumulated in a time-series manner on an object basis, and the accumulated foreign substance map data can be used for automatic recording of selected data and generation of reports, and furthermore, can be used as a dataset for additional training of a deep learning model to be implemented as a continuous learning algorithm that continuously improves the accuracy of foreign substance type classification.

[0111] The embodiments described above may be implemented with hardware components, software components, or a combination thereof. For example, the described device and method may be implemented using one or more computers such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor (DSP), a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, etc.

[0112] Software may include computer programs, code, instructions, or combinations thereof, and may be stored on one or more computer-readable recording media or distributed and stored and executed on computer systems connected to a network. Although embodiments have been described through specific drawings, those skilled in the art may apply various technical modifications and variations based thereon.

[0113] As described above, although the present invention has been explained by limited embodiments and drawings, the present invention is not limited thereto, and it is obvious that various modifications and variations are possible within the scope of the technical spirit of the present invention and the equivalent scope of the claims described below by those skilled in the art to which the present invention belongs. Explanation of the symbols

[0114] 100: Conveyor 200: Camera 300: Lighting section 400: Image processing unit 500: Foreign substance map generation unit 600: Blower control unit A: Shooting area S10: Shooting area setting step S20: Image acquisition step S30: Object candidate extraction step S40: Object location calculation step S50: Object characteristic calculation step S60: Object Feature Extraction Step S70: Foreign substance determination step S80: Foreign substance map generation step S90: Control information output step S100: Data cleansing step S110: Learning process automation steps

Claims

Claim 1 A shooting area setting step for setting a shooting area on the upper part of a conveyor transporting recycled aggregate; an image acquisition step for capturing the recycled aggregate passing through the shooting area as a plurality of consecutive frame images; an object candidate extraction step for extracting object candidates distinguishable from the recycled aggregate by analyzing at least one image characteristic among color, shape, and texture for each shooting area; an object position calculation step for converting the image coordinates of the object candidates into the actual transport coordinate system of the conveyor to calculate the conveyor width direction position and transport direction position of the object candidates; an object characteristic calculation step for calculating object characteristic information including at least one of size information, shape information, color information, and texture information for each object candidate; an object feature extraction step for tokenizing the object characteristic information and, to ensure low latency, extracting information among the object characteristic information whose contribution to foreign substance determination is greater than or equal to a preset standard and transmitting it to a deep learning model; a foreign substance determination step for inputting the object characteristic information into the deep learning model to calculate the type of foreign substance of the object candidate; a foreign substance map generation step for generating a foreign substance map including the conveyor coordinates, type of foreign substance, and object characteristic information for each object candidate; the foreign substance A method for recognizing images of foreign substances in recycled aggregate, comprising: a control information output step for outputting a map to a blower control device; a data refinement step including data collection, refinement, preprocessing, and data transmission for the deep learning model learning process; and a learning process automation step in which the deep learning model is retrained when a preset condition is satisfied using the data refined in the data refinement step. Claim 2 A method for recognizing images of foreign substances in recycled aggregates, further comprising, after the image acquisition step, an image preprocessing step of performing at least one of illumination correction, shadow removal, shake correction, background removal, and conveyor belt area masking on the plurality of consecutive frame images. Claim 3 A method for recognizing images of foreign substances in recycled aggregates according to claim 1, further comprising, after the image acquisition step, a multi-frame tracking step of tracking the position change of the same object candidate in the plurality of consecutive frame images and determining whether the position change matches a reference movement amount corresponding to the conveyor's transport speed. Claim 4 A method for recognizing images of foreign substances in recycled aggregates, further comprising a noise correction step in which, in the multi-frame tracking step, if the position change of the same object candidate deviates from the allowable range of the reference movement amount, the object candidate is determined to be a noise object. Claim 5 In claim 1, the foreign substance determination step calculates the detection reliability for the type of foreign substance along with the type of foreign substance of the object candidate using the deep learning model, and the foreign substance map generation step generates a foreign substance map that further includes the detection reliability for each object candidate. A method for recognizing images of foreign substances in recycled aggregates. Claim 6 In claim 5, the method for recognizing images of foreign substances in recycled aggregates further comprises, after the foreign substance determination step, a scattering difficulty calculation step for calculating a scattering difficulty indicating the degree to which the object candidate is difficult to scatter by airflow using at least one of the foreign substance type, location information, size information, shape information, and detection reliability of the object candidate, and the foreign substance map generation step for generating a foreign substance map including a scattering difficulty for each object candidate. Claim 7 In claim 6, the step of calculating the scattering difficulty determines a preset reference scattering value for each type of foreign substance, and calculates the scattering difficulty by applying a correction coefficient based on at least one of the area, major axis, minor axis, contour complexity, circularity, and aggregate stacking degree of the object candidate to the reference scattering value.

Images