System and procedure for identifying objects and predicting an object class

Abstract

Aspects relate to a method and a system for predicting the class of an object, comprising sensor fusion of an RGB image acquisition device and a thermal map image acquisition device. The system and method receive RGB images and thermal map images associated with the object and identify a first and a second area of ​​interest associated with the object; generate a first segmented image associated with the first area of ​​interest and a second segmented image associated with the second area of ​​interest; and derive a prediction of the object's class using the first and second segmented images. Artificial intelligence and machine learning can be employed in identifying and tracking the object, generating the segmented images, and predicting the object's class.

Description

Technical field

[0001] The revelation relates to a system and a procedure for identifying objects and predicting a class of objects. background

[0002] The consumer-grade waste sorting industry has received increasing attention due to its contributions to developing a more sustainable method for processing recyclable waste. Examples of items sorted can correspond to classes such as plastics, metals, cardboard, paper, and glass. A large part of the plastics sorting process is typically performed manually, raising concerns regarding staffing capacity and sorting quality.

[0003] One solution uses one or more robotic arms to separate metal, plastic, and organic waste into different baskets using an infrared sensor. However, this application does not take into account the different types of plastics (such as PET and HDPE) and is limited in that it only uses a single sensor type to analyze the thermal images generated by an infrared sensor.

[0004] Another solution, disclosed in US patent publication US20180016096A1, focuses on separating recyclable elements from non-recyclable elements and further categorizing the recyclable elements based on an object class such as plastic, paper, metal, cardboard, fabric, and glass. However, this solution is relatively expensive because it uses a large number of sensors to generate digital images for classifying these recyclable elements, and it does not differentiate between various types of plastics.

[0005] Another solution, disclosed in EP patent publication EP0824042A1, uses a laser source that directs a polarized beam at various types of plastics (such as PET, HDPE, PVC), which can be expensive and requires extensive or burdensome safety considerations.

[0006] Another solution, disclosed in US20140367316A1, is an industrial-scale solution rather than a user-friendly one. This disclosure uses terahertz spectroscopy (THz spectroscopy) to identify and differentiate industrial and sheet plastics (black HIPS, PC, PS, ABS, PC-ABS, acrylic, and acetal), non-recyclable plastics (especially PET and HDPE bottles), and elemental-level sorting (any 3D shape). This system requires complex pre-sorting and a long waste stream chain, and demands a substantial investment; that is, this spectroscopic THz solution attempt is relatively slow and expensive.

[0007] While near-infrared (NIR) and RGB camera-based sensors are among the most common sensors used in residential waste sorting, each has limitations and can only provide reliable predictions under specific conditions. For example, NIR cannot penetrate plastics and cannot obtain color information from objects. Additionally, NIR optical sorters cannot sort labels on plastic bottles, black plastics, and multilayer plastics. Poor sorting performance can also result from cross-contamination with food residues. RGB cameras are limited by their sensitivity to varying lighting conditions, optical illusions caused by overlapping elements, and their inability to detect and differentiate bottle-type objects covered with labels or irregularities such as dirt and dust.

[0008] There is a need to create a solution to mitigate at least one of the problems mentioned above. Summary

[0009] The disclosure provides a relatively fast and intuitive automated object identification and class prediction based on a sensor fusion solution. The solution uses a power intensity heat map image generated from electromagnetic THz radiation and RGB images to classify a specific plastic stream (e.g., PET and HDPE bottles) for small-scale or consumer-grade MRF and recycling companies, creating a novel combination of high-contrast heat mapping, spatially high-resolution imaging (for RGB images), and intuitive machine learning analysis. Aspects of the disclosure include a unique sensor fusion (RGB and THz sensors) and its data-driven (feature-driven) machine learning algorithm to enable a waste sorting solution.

[0010] According to the present invention, a method for predicting the class of an object according to claim 1 is provided. A system for predicting the class of an object according to the invention is defined in claim 7.

[0011] The dependent claims define some examples that are assigned to the process or the system. Brief description of the drawings

[0012] The invention is better understood when viewed in conjunction with the non-limiting examples and the accompanying drawings, which show: - Fig. 1. A flowchart of a procedure for identifying an object and predicting a class of the object; - Fig. 2A a system design for identifying objects and predicting an object class; Fig. 2B an example of the images taken by a first sensor and a second sensor; - Fig. 3A to Fig. 3E different image processing modules and techniques that are assigned to the identification of objects and the prediction of an object class; - Fig. 4A a process flow for predicting an object class; Fig. 4B, an accuracy test result of the prediction model according to certain embodiments; - Fig. 5 an example of a summary module that can be operated to combine multiple forecast results to form a final forecast result; and - Fig. 6 a schematic representation of a processor 210 for identifying objects and / or predicting an object class according to certain embodiments. Detailed description

[0013] The following detailed description refers to the accompanying drawings, which illustrate specific details and embodiments in which the disclosure can be put into practice. These embodiments are described in sufficient detail to enable those skilled in the art to put the disclosure into practice. Other embodiments may be used, and structural and logical modifications may be made without departing from the scope of the disclosure. The various embodiments are not necessarily mutually exclusive, since some embodiments may be combined with one or more other embodiments to form new embodiments.

[0014] Embodiments described in the context of one of the systems or methods are valid analogously for the other systems or methods.

[0015] Features described in the context of one embodiment may be applicable to the same or similar features in other embodiments. Features described in the context of one embodiment may be applicable to other embodiments even if they are not explicitly described in those other embodiments. Furthermore, additions, combinations, and / or alternatives described for a feature in the context of one embodiment may be applicable to the same or a similar feature in other embodiments.

[0016] In the context of some embodiments, the articles “a”, “an” and “the”, which are used with regard to a feature or an element, contain a reference to one or more of the features or elements.

[0017] As used herein, the term “and / or” includes any combination of one or more of the associated listed elements.

[0018] As used here, the term "object" includes any object, in particular a recyclable or reusable object, that can be sorted according to type and / or class. For example, plastic objects can be sorted according to whether they belong to the high-density polyethylene (HDPE) or polyethylene terephthalate (PET) class. The objects may include bottles, jars, containers, sheets, trays, etc., of various shapes and sizes.

[0019] As used here, the terms "assign," "associated," "assign," and "assign" indicate a defined relationship (or cross-reference) between at least two elements. For example, an image assigned to an object may contain a defined area of ​​interest that focuses on the object for further processing using object detection and / or one or more tracking algorithms.

[0020] As used here, the term "module" refers to an application-specific integrated circuit (ASIC); an electronic circuit; a combinational logic circuit; a field-programmable gate array (FPGA); a processor (shared, dedicated, or a group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-a-chip, or forming a section thereof, or containing them. The term "module" may include memory (shared, dedicated, or a group) that stores code to be executed by the processor.

[0021] As used here, the term "RGB image" refers, simply put, to images that include image data containing colors encoded as three numbers assigned to the colors 'red', 'green' and 'blue'.

[0022] Fig. Figure 1 shows a flowchart of a procedure 100 for predicting the class of an object, comprising the following steps: receiving an RGB image and a power intensity image (which may be in the form of a heat map image) associated with the object (step S102); identifying, in the RGB image and the heat map image, a first area of ​​interest and a second area of ​​interest associated with the object (step S104); obtaining a first segmented image associated with the first area of ​​interest and a second segmented image associated with the second area of ​​interest (step S106); and obtaining a prediction of the object's class using the first segmented image and the second segmented image (step S108). The step of obtaining a prediction of the object's class may include a step of pairing the first segmented image and the second segmented image.The pairing process may include steps of assigning a common identifier to the first segmented image and the second segmented image.

[0023] Fig. Figure 2A shows a setup of a system 200 for predicting classes of objects according to method 100. The system 200 for predicting a class (e.g., HDPE plastic, PET plastic, paperboard, metal glass, etc.) of an object 201 comprises a first sensor 202 configured to acquire at least one RGB image 204 associated with the object; and a second sensor 206 configured to acquire at least one power intensity image, i.e., a thermal map 208, associated with the object 201. The object 201 may be positioned on a platform 205, such as a conveyor belt. The system 200 includes a processor 210 configured for data or signal communication with the first sensor 202 and the second sensor 206 to receive the at least one acquired RGB image 204 and the thermal map image 208.It is intended that multiple RGB images and multiple heat map images of the same object are acquired at different timestamps, with each image pair (RGB, heat map) potentially yielding a predicted class of the object. In certain embodiments, multiple predicted classes associated with the image pairs can be combined to derive a final class prediction.

[0024] The first sensor 202 can be an RGB recording device, such as a camera, video recorder, or camcorder, to continuously capture RGB images of the object at various positions on the platform 205. The second sensor 206 is a terahertz-based (THz-based) image acquisition device configured to capture thermal map images of the object at various positions on a platform. The THz-based image acquisition device can include a THz source configured to emit electromagnetic radiation in a THz, super-THz, or sub-THz frequency range. For example, a sub-THz range (0.1–0.7 THz) and a higher frequency range (1.0–3.0 THz).The THz-based image acquisition device may also include a scanning device designed to receive electromagnetic radiation (emitted by the THz source) incident on an object and to generate a power intensity heat map image.

[0025] In certain embodiments, the first sensor 202 and the second sensor 206 can be operated remotely to capture images of the object simultaneously or overlapping at predetermined intervals. In certain embodiments, the first sensor 202 and the second sensor 206 can be configured to continuously capture or record their respective images.

[0026] Fig. Figure 2B shows an example of a captured RGB image 204 and a thermal map image 208 (also referred to as a THz output image). In certain embodiments, multiple RGB images of the object (known as frames) can be captured and stored in an associated database of the first sensor 202. The captured THz thermal map image frames within a specific THz spectrum can be stored in an associated database of the second sensor 206 and converted into multiple frames of thermal map images 208.

[0027] The acquired images 204, 208 can be sent to processor 210 for further processing. The processing of one or more acquired thermal map images may include the step of obtaining image data via Fourier transform spectroscopy and / or THz time-domain spectroscopy techniques, as is known to those skilled in the art.

[0028] In certain embodiments, as in Fig. As shown in Figure 3A, the processor 210 comprises an image labeling module 212, an object detection and tracking module 214, an image segmentation module 216, a pairing and classification module 218, and a summarization module 220. Modules 212, 214, 216, 218, and 220 are described below for clarity and ease of understanding. It is intended that one or more of the aforementioned modules can be combined. It is further intended that all modules can be combined.

[0029] The image labeling module 212 is operable, allowing any image to be labeled with a graphical annotation tool and saved as a file in an extensible markup language (XML file). In certain embodiments, the images were marked to belong only to a single class, "Object." The annotation can be in the form of bounding boxes 302 and 304 around the objects of interest. Fig. Figure 3B shows an example of bounding boxes 302 annotated around objects of interest in an RGB image 204, and of a bounding box 304 around an object of interest in a THz heat map image 208.

[0030] The object detection and tracking module 214 comprises an object detection algorithm and a tracking algorithm to receive image data from one or more RGB images and image data from one or more thermal map images for analysis. This allows the module to obtain the area of ​​interest (ROI) of each object at different timestamps, i.e., images of the same object tracked over a period of time. The object can be positioned on a moving platform, such as a conveyor belt, moving at a predetermined speed in the range of 0.45 to 1.5 meters per second.

[0031] In certain embodiments, the object detection algorithm may be a machine learning algorithm with pre-trained deep learning or an artificial intelligence-based algorithm such as, but not limited to, a deep neural convolutional network model such as the YOLO (You Only Look Once) v4 model, which uses a DenseNet classification model as its basis.

[0032] In certain embodiments, the tracking algorithm can be a feature-matching-based algorithm to track objects across multiple image frames for both RGB and THz-based image capture devices. An example of such a tracking algorithm is the DeepS-ORT algorithm. This algorithm checks the object's key features in one frame and compares them in subsequent frames to track the object along the conveyor belt.

[0033] In certain embodiments, the object detection algorithm and the object tracking algorithm can be combined or cascaded. In the combined configuration, both algorithms are configured to receive image data from the image acquisition devices 202 and 206. The tracking algorithm is configured to receive an output from the object detection algorithm (i.e., bounding boxes defined as the ROI) as an input parameter for preprocessing. The combined output of the object detection algorithm and the object tracking algorithm is an output video showing the ROI around each captured image object across all image frames.

[0034] The image segmentation module 216 is operable to receive the output video data from the module 214 as input and to generate segmented images of the RGB and THz images as output based on the ROI identified at each image frame. In other words, as the object is tracked across the conveyor belt's movement, a segmented RGB image and a segmented heat map image for each object are stored in a database at a specific timestamp. The image segmentation module 216 can include a segmentation algorithm to model segmentation as a "max-flow / min-cut" optimization problem to be solved by a "grab-cut" algorithm that iteratively extracts the foreground of an image (i.e., the ROI) using graph slices.The algorithm can use a Gaussian mixing model (GMM) to model the foreground and background of the image based on a color distribution of the target object (according to the definition of the ROI) and that of the background, in order to obtain the segmented foreground image of the objects on the platform. An example of the extracted segmented images 306, 308 in the form of "GrabCut" images for RGB (segmented image 306) and THz (segmented image 308) is shown in [reference missing]. Fig. 3C shown.

[0035] The pairing and classification module 218 is operable to process the segmented images output by the image segmentation module 216. Several image processing procedures can be implemented on the segmented images, including, but not limited to, (i) adding a background (e.g., a black screen), (ii) enlarging, and (iii) normalizing. The addition of a black screen is motivated by the requirement to resize the input dimensions of the segmented RGB image and the corresponding heat map image, which was captured at the same timestamp, such that all RGB and THz images are the same size.To minimize distortion and deformation of the segmented images 306, 308 caused by resizing the image, a black canvas with a fixed width and height was added in the background for each image, after which the THz input image 312 is 356x371 pixels and the RGB input image 310 is 350x480 pixels, as shown in . Fig. It is shown in 3D.

[0036] The extension can be based on a tensor stream-based extension, which is applied to eliminate the effect of both the position and orientation of objects while still maintaining the black background color. Inverted or rotated images for input images 310 and 312 can be generated as additional input images. Fig. 3E shows the additional inverted or rotated images 314, 316, which were generated for the RGB input image 310 and the THz input image 312, respectively.

[0037] Normalization can be performed on all input images 310, 312, 314, 316 before classification takes place. Each pixel value in an image can be divided by a reference value (e.g., 255), which ensures that all values ​​are in the range of 0 to 1.

[0038] The classification module 218 is also operable to fuse or combine the processed input images 310, 312, 314 and 316 using one or more deep neural convolutional networks with multiple inputs. Fig. Figure 4A shows a process flow of the classification process 400. In certain embodiments, the one or more multi-input deep convolutional neural networks comprise two pre-trained convolutional neural network models 402, 404 with deep learning capabilities for large-scale image recognition (such as VGG16 models) for feature extraction from the input images 310, 312, 314, and 316. In certain embodiments, a shared layer 406 and one or more dense layers 408 are configured to receive and concatenate the extracted features to generate a single prediction of the object class 410. In certain embodiments, the deep learning is modeled as an error minimization problem with a loss function, such as a categorical crossentroy loss function, which is applied to an optimization device, such as an Adam optimization device, for training.As the optimization progresses, the parameters are updated based on forward / backward propagation techniques until the error reaches a minimum value or is below an acceptable reference value (i.e., optimized or nearly optimized).

[0039] Although it is possible to provide a final prediction based on a single predicted outcome, multiple predictions of the same object using several processed images reduce the risk of an incorrect prediction that could be caused by improperly captured images (e.g., blurry images). The Summary Module 220 works to combine multiple predicted outcomes associated with each object to be classified.

[0040] As in Fig. As shown in Figure 5, for each object (including an assigned identifier id1), three pairs of segmented images (502, 504, 506) are forwarded to the classification module 218. Raw images, corresponding to the three pairs of segmented images 502, 504, 506, are acquired by the first sensor 202 and the second sensor 206 at three different times t1, t2, and t3. Each object to be classified is predicted at three times to generate a result 1, a result 2, and a result 3. The three results are aggregated to produce the final result. In certain embodiments, the aggregation can take the form of voting. In other words, the most frequently predicted class will be the object's final result and will be stored in the database with the respective tracking identifier id1. It is envisaged that further statistical methods can be used for aggregation.

[0041] It is worth noting that the RGB camera 202 and the THz sensor 206 are synchronized to capture input video data and thermal map data, respectively, into the processor 210. After the data is captured, the processor 210 performs object detection to locate the object and tracks it across frames (at different timestamps). For this process, the processor 210 stores a segmented RGB image and a segmented THz image for each object at each timestamp in a database. These images are then fed into the hybrid classification model, which predicts the class.

[0042] It is being considered that the machine learning and / or artificial intelligence algorithms used in various modules could undergo training and testing before being deployed operationally. Such training could include the creation of a test and training dataset, as well as supervised and unsupervised learning.

[0043] In certain embodiments, after data collection by the first sensor 202 and the second sensor 206, the image frames are extracted from the video files at 30 frames per second (FPS). After extraction, 535 RGB and 535 corresponding THz images may be present, totaling 1070 images. The extracted images can then be labeled using the image label module 212, and the annotations are saved as XML files. After the images have been labeled, the data set is split into a training and a test data set. The training data set consists of 85% (910 images) of the data, and the test data set consists of 15% (160 images) of the data. After splitting, the training data set can be extended (this extension is different from that performed by the pairing / classification module 218).The extension includes horizontal and vertical flipping, 90-degree rotations, 10% cropping, 1.5px blurring, etc. After the extension, the training dataset contained 2727 images. Based on these 2727 images, 3545 tagged objects were identified. This dataset is then used to train the object detection and tracking module 214.

[0044] In certain embodiments, the extension performed on the segmented images by the pairing and classification module 218 includes the generation of training and validation images for training and validation. For example, after flipping and rotating, 3120 training images and 634 validation images can be generated for both THz and RGB, as shown in Fig. 3E is shown to be generated to train the hybrid model.

[0045] In certain embodiments, a total of 256 test images are available, of which 81% are correctly predicted, as shown in the f1 score column below the accuracy row. Fig. 4B is shown. The accuracy increases to 0.84 (84%) and 0.93 (93%) in the validation and training datasets.

[0046] It is envisaged that further computer vision / image processing techniques known to a person skilled in the art can be combined / supplemented to form further embodiments to complement and / or replace the ML / AI algorithms. For example, a multi-stage feature detector can be designed instead of a single-stage feature detector.

[0047] In certain embodiments, the processor 210 may include hardware components, such as one or more server computers, configured in a distributed or non-distributed configuration to implement modules 212, 214, 216, 218, and 220. These hardware components may be complemented by a database management system configured to create one or more industry-specific characteristic databases. In certain embodiments, the industry-specific characteristic databases may include analysis modules to relate one or more data sets to an industry. Such analysis modules may include an expert rule database, a fuzzy logic system, or other artificial intelligence modules.

[0048] Fig. Figure 6 shows a server computer system 600 according to one embodiment. The server computer system 600 includes a communication interface 602 (which is configured, for example, to receive captured images from one or more image capture devices 206). The server computer 600 further includes a processing unit 604 and a memory 606. The memory 606 can be used by the processing unit 604 to store, for example, data to be processed, such as data associated with the captured images, intermediate results output by modules 212, 214, 216, 218, and / or final results output by module 220. The server computer is configured to execute the procedure of Fig. 1. It should be noted that the Server Computer System 600 can be a distributed system containing multiple computers.

[0049] In certain embodiments, a computer-readable medium is provided containing program instructions which, when executed by one or more processors, cause the one or more processors to perform the procedure according to the embodiments described above. The computer-readable medium may include a non-transient computer-readable medium.

[0050] In certain embodiments, the ML / Al algorithms may include algorithms such as neural networks, fuzzy logic, evolutionary algorithms, etc.

[0051] It must be acknowledged that the use of electromagnetic radiation operating in the THz frequency range (THz radiation) to generate thermal map images is complementary to RGB images, and vice versa. Thermal map images generated using the THz radiation source can capture pronounced image data related to displacement, vibration, and rotation according to molecular structure due to the sensitivity of THz radiation, thus enabling molecular spectroscopy in materials investigation. However, such sensitivity of THz radiation can unduly influence the resulting thermal map image for an object if the object undergoes dimensional changes.For example, expansion changes made to the same object, including but not limited to changes in one or more shapes, sizes, volumes, thicknesses of layers or walls, curvatures, and forms, result in heterogeneous expansion differences, which can unduly affect how the object appears in the heat map view. Some such expansion changes may be caused by external stresses / forces applied to the object, such as distortion, compression, and deformation, resulting in various states of the object, such as a flattened state, a deformed state, a compressed state, a crushed state, a torn state, etc.This results from the properties of objects, which in turn can influence the prediction of an object class. Since these conditions may not be visible in an RGB camera, the image data obtained from RGB images can be paired and combined with the image data obtained from the thermal map images generated by the THz radiation source to achieve higher prediction precision / accuracy.

[0052] While the disclosure has been specifically shown and described with reference to specific embodiments, it should be clear to those skilled in the art that various changes to the form and details therein can be made without departing from the concept and scope of the invention as defined in the accompanying claims. The scope of the invention is thus defined by the accompanying claims, and therefore it is intended that all modifications that fall within the scope and equivalence of the claims are included. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 20180016096A1

[0004] EP 0824042A1

[0005] US 20140367316A1

[0006]

Claims

[1] A method for predicting a class of an object, comprising the following steps: (a) receiving an RGB image and a heat map image associated with the object; (b) identifying in the RGB image and in the heat map image a first region of interest and a second region of interest associated with the object; (c) obtaining a first segmented image associated with the first region of interest and a second segmented image associated with the second region of interest; and (d) obtaining a prediction of the class of the object using the first segmented image and the second segmented image; wherein the step of obtaining the prediction of the class of the object comprises a step of pairing the first segmented image and the second segmented image. [2] The method of claim 1, further comprising the steps of repeating a predetermined number of steps (b) to (d), storing the predicted object class associated with each pair of a first segmented image and a second segmented image, and summarizing the predicted results to determine a final prediction. [3] The method of claim 2, wherein the step of summarizing the ranked results to determine the final ranked result comprises a step of assigning the most frequently predicted object class to be the final prediction. [4] A method according to any one of the preceding claims, wherein the step of obtaining a prediction of the object class further comprises a preprocessing step of resizing the first segmented image and the second segmented image. [5] A method according to any one of the preceding claims, wherein the step of obtaining a prediction of the object class further comprises a step of concatenating feature data associated with the first segmented image and the second segmented image. [6] The method of claim 5, further comprising the step of passing the concatenated data through a pre-trained neural network. [7] System for predicting a class of an object, comprising: a first sensor configured to capture at least one RGB image associated with an object; a second sensor configured to capture at least one heat map image associated with the object; a processor configured to be in data or signal communication with the first sensor and the second sensor to: identifying in the RGB image and the heat map image a first region of interest and a second region of interest associated with the object; Obtaining a first segmented image associated with the first region of interest and a second segmented image associated with the second region of interest; and Obtaining a prediction of the class of the object using the first segmented image and the second segmented image; wherein the prediction of the object class includes concatenating feature data of the first segmented image and the second segmented image. [8] The system of claim 7, wherein the first sensor is an RGB imager and the second sensor is a terahertz (THz) imager. [9] The system of claim 8, wherein the RGB image capture device and the THz image capture device are configured to obtain a plurality of RGB images and corresponding heat map images of the object in a predetermined period of time. [10] The system of claim 9, wherein the processor is configured to store the predicted classes of the object associated with each pair of the plurality of RGB images and heat map images as intermediate results, and the processor is further configured to aggregate the predicted class to derive a final predicted result. [11] The system of claim 10, wherein the final predicted result is derived based on assigning the most frequently predicted object class to be the final predicted result. [12] The system of any one of claims 7 to 11, wherein the processor is configured to perform: resizing each pair of an RGB image and a heat map image and / or passing the concatenated feature data through a pre-trained neural network.

Images