Method and system for characterizing one or more substances - Patents.com

Abstract

A method and system for characterizing one or more materials includes: scanning one or more materials with a perception system including an X-ray sensor configured for multi-energy imaging to acquire at least lower-energy X-ray images and higher-energy X-ray images; segmenting the images acquired by the perception system; and determining, for each segmented object, data indicative of area density and data indicative of atomic number by analysis of the lower-energy X-ray images and the higher-energy X-ray images, the data indicative of area density and atomic number determined by a calibrated model; calculating, for each segmented object, data indicative of mass based on the data indicative of area density and the data indicative of area for the segmented object; providing the data indicative of atomic number as input to a trained neural network, which is configured to label each segmented object, and combining the data indicative of mass with each of the labeled segmented objects.

Description

[Technical field]

[0001] The present invention relates to a method and system for characterizing one or more materials. The present invention further relates to a computer program product. Furthermore, the present invention relates to a recycling device configured to sort a waste stream. [Background technology]

[0002] Characterization of one or more materials has several important applications, e.g., in recycling processes, agricultural processes, food production processes, etc. The characterization can be used, e.g., for quality control, valuation, and process engineering and control. For example, in waste treatment, traditionally, many waste streams are recycled suboptimally due to a lack of characterization data. There is a need for suitable characterization techniques for material streams, e.g., bulk solid waste streams.

[0003] The conventional approach to waste characterization is still typically manual inspection of objects by a person, e.g., factory personnel working in a special facility. This approach is slow, subjective, expensive, and ultimately conveys little information about the particles in the waste stream. In some conventional methods, samples are taken and inspected / analyzed, e.g., in a laboratory. This process can be too time consuming (chemical analysis can take days, weeks, or months) and can result in increased costs. Furthermore, only a small fraction of the total amount of material / objects is characterized. Typically, material quality is difficult to measure, so many material streams are recycled suboptimally.

[0004] There is a need for fast, objective and / or automated methods of conveying data at a more detailed level. Summary of the Invention [Problem to be solved by the invention]

[0005] SUMMARY OF THE DISCLOSURE It is an object of the present invention to provide a method and system which obviates at least one of the above mentioned drawbacks.

[0006] Additionally or alternatively, it is an object of the present invention to provide improved material characterization of one or more materials.

[0007] Additionally or alternatively, it is an object of the present invention to provide improved material characterization. [Means for solving the problem]

[0008] To that end, a method of characterizing one or more materials includes scanning the one or more materials with a perception system having an X-ray sensor configured for multi-energy imaging to obtain at least a lower energy X-ray image and a higher energy X-ray image; segmenting the images obtained by the perception system to isolate one or more distinct objects within the images, where data indicative of an area of ​​the segmented objects is determined; and for each of the segmented objects, determining data indicative of an area density and data indicative of an atomic number by analysis of the lower energy X-ray image and the higher energy X-ray image, where the data indicative of the area density and the atomic number are determined. the data indicative of the area of ​​each of the segmented objects is determined by a model that is calibrated by performing multi-energy x-ray imaging with different materials having known area densities and atomic numbers; calculating, for each of the segmented objects, data indicative of a mass based on the data indicative of the area density and the data indicative of the area for each of the segmented objects; and providing, for each of the segmented objects, the data indicative of at least the atomic number as input to a trained neural network, wherein the trained neural network is configured to label each segmented object, wherein the data indicative of the mass is assigned to each of the labeled segmented objects. Associated The method further comprises:

[0009] Objects in the material stream can be identified and characterized. Characterization of each individual object may provide different characterization parameters. This allows to distinguish objects in different material classes. Additionally or alternatively, a mass balance can be determined. Such derived information can be used for post-modeling using the physical properties of the characterized objects. Additionally or alternatively, certain properties of each segmented object can be determined, such as geometry, diameter, shape, chemical properties, etc.

[0010] The present invention can effectively enable analyzing heterogeneous material flows. This can be done without requiring extensive resources (workload, time, etc.). Even the mass balance of the heterogeneous material flow can be efficiently determined, while the objects and / or materials within the material flow have not yet been sorted.

[0011] The present invention provides significant advantages in sorting material streams, e.g. waste streams. However, the present invention can be used in a variety of other applications besides material sorting. For example, the present invention can be used to determine the value of a material stream. By identifying the characteristics of different components within the material stream, the value can be estimated. Sampling a material stream (e.g. waste stream) can take too much time and effort. A system according to the present invention allows for determining mass balance and value estimation.

[0012] The mass can be estimated based on an X-ray model using X-ray images with at least one energy level. The areal density is mainly determined by the higher energy absorption rate. By taking into account only the higher energy absorption rate, it is possible to estimate the mass without needing X-ray images of more than one energy level (see lower energy level X-ray image and higher energy level X-ray image). The estimate can be provided by considering only one X-ray energy range. However, in practical cases, at least two X-ray energy levels can be adopted.

[0013] The mass can be predicted based on an X-ray model using X-ray images with at least two different energy levels (it is also possible to make an estimation based on a single X-ray image (one energy level)). It will be appreciated that multi-spectral X-ray imaging with more than two X-ray energy levels can also be employed. The X-ray images can be acquired by the same X-ray imaging device. However, it is also possible to use multiple X-ray imaging devices.

[0014] In some examples, a lower energy X-ray image and a higher energy X-ray image of the one or more materials (e.g., material stream) are taken, and a model is used to calculate area density and atomic number from the two X-ray images of the one or more materials. The mass can be calculated based on the sum of all determined area densities. The mass of a segmented object can be calculated directly from the density based on the sum of all pixels of the segmented object. The atomic number provides additional information regarding the chemical composition of the material. Knowledge of the atomic number of each individual segmented object / component can provide a significantly improved material characterization that takes into account the chemical composition. The atomic number may not be required to calculate the mass of the individual segmented objects / components.

[0015] The area density can be viewed as mass per area, and the mass can be determined based on the area and the area density. An area of ​​each of the segmented objects can be identified based on the segmentation in the imaging data.

[0016] A lower energy X-ray image and a higher energy X-ray image may be taken to provide information about two ranges in the spectrum. In some examples, the at least two X-ray images may be taken simultaneously. In X-ray imaging, at least two values ​​may be obtained for each point in space (at least the lower energy and the higher energy). In some examples, the X-ray sensors are positioned on top of each other. An upper sub-sensor may sense the lower energy and a lower sub-sensor may sense the higher energy. In some examples, multi-energy X-ray imaging may be performed such that each pixel is associated with more than two values. In some examples, each pixel may be associated with two values, the lower energy and the higher energy for dual energy X-ray imaging.

[0017] Advantageously, the present invention may provide improved labeling using a trained artificial neural network. In this way, a better determination of the analyzed object / component of one or more materials can be obtained. For example, a better determination of the material composition of a material stream can be achieved. The labels of the segmented objects can also be considered as additional characterization data that can be presented to a user, for example by a generic user interface. In some examples, the determined data is used to draw a virtual model (digital twin of the one or more materials, e.g., material stream), enabling virtual experiments or modeling. In some examples, the determined features / characteristics of the segmented objects are generic. The labels can provide more application-specific information (for example, labels for battery classification, i.e., wood and plastic, pine, oak, plastic, ferrous materials, non-ferrous materials, minerals, electrical components, scrap metal, concrete, etc.). It will be appreciated that the data can be used in different ways and that many different uses are anticipated. For example, the labels can also be price, or quality type, chemical composition and / or content, etc.

[0018] The characteristics / properties can be considered as measurable quantities (allowing for objective determination). The labels can take into account subjective considerations (e.g., price, perceived quality, maturity, etc.). The invention can be employed for a multi-variety of mixtures or streams of matter in which individual components / objects are to be differentiated from one another. Optionally, the mixtures or streams of matter contain at least one type of material in which a distinction is to be made.

[0019] Different algorithms can be used to provide segmentation of objects / components in the image. For example, the method may include an object detection algorithm. Examples of object detection algorithms are thresholding, watershed, semantic segmentation or instant segmentation by neural networks. Other algorithms are also possible to segment the objects / components. The segmented objects can be cut out and evaluated individually for feature / characteristic extraction.

[0020] Optionally, the segmented objects are individually processed to provide a classification using the trained neural network. This operation may also be performed at least partially in parallel. It is also anticipated that multiple trained neural networks may be used.

[0021] Optionally, the X-ray sensor is a dual-energy X-ray sensor. Dual-energy X-ray imaging can be cheaper than multi-energy X-ray imaging. However, it is also possible to use a multi-energy X-ray sensor in the X-ray sensor, because in this way more accurate results can be obtained.

[0022] Optionally, the perception system further comprises a depth imaging unit for determining data indicative of a volume of the segmented object.

[0023] Different types of depth imaging units can be used. Volumetric values ​​can be an important metric for characterizing one or more materials.

[0024] Optionally, the depth imaging unit comprises at least one of a 3D laser triangulation unit or a 3D camera.

[0025] Other types of depth imaging units are also possible.

[0026] Optionally, the perception system further comprises a colour imaging unit arranged to capture colour images of the segmented object.

[0027] Multiple sensors may be used to sort the waste. In some examples, a first sensor is used (e.g., employing X-rays) to obtain, among other things, an indication of area density, and a second sensor is used (e.g., 3D camera, laser triangulation, etc.) to obtain, among other things, an indication of volume. Other sensors, such as color cameras, may also be used to gather more information. In an advantageous embodiment, multiple pre-processing steps are performed before the recorded sensor data is fed to a trained neural network. The images are aligned and the detected objects are segmented. Aligning the images may involve at least one of shifting, scaling, shearing, changing viewpoint, etc. Aligning the images may involve any other distortion correction to the images to obtain an aligned image. Multiple properties, such as mass, volume, atomic number, density, shape, etc., are determined for each object. This data is then used as input to the neural network. This allows neural networks to be obtained that are better able to distinguish between different materials and therefore waste groups in waste recycling applications. More efficient neural networks can also be obtained in this way.

[0028] Optionally, data from different subsystems of the perception system are aligned prior to determining characteristic features for each of the segmented one or more objects.

[0029] In some examples, X-ray imaging is used to determine the mass and atomic number for each of the segmented objects. However, other sensors may also be used for imaging. Optionally, images from different sensors are aligned for data fusion. For example, the sensors and / or detectors for imaging may have different fields of view, different distortions, magnifications, etc. Alignment can be performed by performing an alignment algorithm or a transformation algorithm on the imaging data received from the multiple sensors / detectors.

[0030] In some examples, data from different imaging sensors / detectors may be fed into a 3D matrix with multiple "image layers", e.g., red-green-blue image data, lower energy x-ray, higher energy x-ray, height (3D camera), etc. Additional data may also be included, e.g., derived images related to atomic number, surface density images, density (e.g., by combining volume and surface density, density can be determined for each segmented object / component).

[0031] Optionally, for each of the segmented one or more objects, further characteristic features related to at least one of volume, size, diameter, shape, texture, color, or eccentricity are determined. Other characteristic features of object pixel values ​​of the different sensor images, such as X-ray absorptivity, height, color, etc., as well as derived images, such as atomic number, area density, density, chemical composition, and gradient, such as mean, standard deviation, third or higher moments, histograms, etc., can also be determined. Some exemplary shape parameters that can be determined are circumference, solidity, area, convexity area, convexity, circularity, straightness, curliness, sphericity, aspect ratio, ferret diameter, equivalent diameter, minor and major axis lengths, maximum and minimum chord lengths, other diameter definitions. It will be appreciated that any combination of the preceding characteristic features is also possible.

[0032] Since multiple properties are determined for each of the segmented objects, the objects can be positioned in a multi-dimensional space, which can provide an enhanced overview of a material stream, for example, via a generic user interface of a computer program product that presents the determined properties of the one or more materials.

[0033] Optionally, the one or more materials are moved on a conveyor forming a material stream, where the material stream is scanned by the perception system for characterization of objects within the material stream.

[0034] In some examples, first the objects (e.g., components, particles) in the material stream are segmented. In this way, the objects can be easily cut out, separated and / or separated. The objects can be separated even when they are in contact with each other. In a next step, a set of features / characteristics of each particle can be identified. These features / characteristics can then be fed into a machine learning model, e.g., an artificial neural network.

[0035] The mass, volume, diameter, shape, texture, contour, etc. of particles can be determined on the fly or in real time. The mass of the object can be calculated using non-contact measurements (see X-ray). The mass of the individual objects can be calculated using a physical model based on X-ray physics. This can be considered as a pre-processing step before feeding the imaging data to a neural network for classification. Other techniques can also be used to determine the mass of the individual objects. In some examples, machine learning models are employed.

[0036] Optionally, characteristic features of the segmented one or more objects are stored to construct a digital twin model.

[0037] The present invention allows for component level characterization of material streams. In this way, a digital twin model can be set up. Virtual testing and simulations can then be performed using the digital twin model.

[0038] A physical material stream can be scanned and converted into a virtual representation, also called a "digital twin". This digital counterpart includes the physical properties of all particles in the stream, such as mass, volume, density, shape, size, material, visual appearance, texture, etc., as well as the original images taken by different cameras. The digital twin enables virtual experiments (e.g., performing sieving experiments based on computer data in a lab without getting your hands dirty), particle-based process simulations, and provides an accessible and intuitive way to investigate material streams and gain new insights.

[0039] Optionally, the one or more substances are characterized prior to transportation to determine a first digital identification marker, and wherein following transportation to a remote location the one or more substances are characterized to determine a second digital identification marker, and wherein the first digital identification marker and the second digital identification marker are compared to each other to determine a change in content during transportation.

[0040] In some instances, the container or cargo contains heterogeneous material. The cargo can be transported over long distances and re-characterized. Digital fingerprinting can be done at a detailed level, since a detailed characterization can be provided. If this is done at the origin and destination, it can be determined if the container or cargo has been tampered with. It can be checked if the container / cargo including its contents is the same. Furthermore, details on what has been added or removed can be identified.

[0041] Optionally, the one or more materials are heterogeneous.

[0042] It will be appreciated that a material stream can be defined as a flow of material. In some instances, a stream can be a particular material category that is diverted in a particular way. In some instances, a stream can be a mixture of several material categories that are diverted in a particular way. Examples of material streams include decomposed materials sent to a reuse market, mixed waste sent to a mixed waste recycling facility, etc.

[0043] Optionally, the one or more materials are selected from the group consisting of solid waste, manufactured products or components, agricultural products, or batteries.

[0044] The present invention offers significant advantages when applied to the sorting of waste batteries. The batteries may have the same appearance (e.g. from the outside, several batteries often look the same). A stream of batteries, including different battery types, can be guided along a conveyor belt. Different types of batteries can be effectively distinguished from one another using the method according to the present invention. For example, a distinction can be made between alkaline and lithium-ion batteries. Furthermore, characteristics such as size, shape, configuration (e.g. battery pack) can also be used to distinguish between the different batteries in the battery stream. It is also possible to identify what materials are contained in the identified / segmented batteries, for example which cathode material (e.g. cobalt-nickel, etc.) is contained in the battery.

[0045] In some examples, the method is used for characterization of agricultural processes. For example, a current of pears (or other fruits) can be characterized. A first feature can be defined as a value indicating the (relative) amount of small dots on the outer surface of the pear, a second feature can be defined as a value indicating how many large spots are present on the outer surface of the pear, a third feature can be defined as a value indicating the volume of the pear, etc. A label can then be defined, for example, as the commercial value of the individually segmented pears. In this example, it will be appreciated that other features / characteristics and / or labels can also be defined. Other applications are also possible, for example for characterization of food streams.

[0046] In some examples, the method may include at least some of the following steps: imaging one or more materials (e.g., X-ray scanning) to obtain at least one image; segmenting and clipping components from the image; and identifying features and / or characteristics for each segmented component, such as mass (e.g., based on dual-energy X-ray imaging or multi-energy X-ray imaging), volume (e.g., based on 3D images), atomic number, shape, eccentricity, size, etc.

[0047] Optionally, a direct in-line characterization system for the bulk solid waste stream is provided.

[0048] According to one aspect, the invention provides a system for characterizing one or more materials, comprising a perception unit arranged to scan the one or more materials, wherein the perception unit comprises an X-ray sensor configured for multi-energy imaging to obtain at least a lower energy X-ray image and a higher energy X-ray image; wherein the system performs segmentation of an image obtained by the perception system to isolate one or more distinct objects within the image, wherein data indicative of an area of ​​the segmented objects is determined; and determining, for each of the segmented objects, data indicative of an area density and data indicative of an atomic number by analysis of the lower energy X-ray image and the higher energy X-ray image. wherein the data indicative of the area density and atomic number are determined by a model that is calibrated by performing multi-energy X-ray imaging with different materials having known area densities and atomic numbers; calculating, for each of the segmented objects, data indicative of a mass based on the data indicative of the area density and the data indicative of the area for each of the segmented objects; and providing, for each of the segmented objects, the data indicative of at least the atomic number as input to a trained neural network, wherein the trained neural network is configured to label each segmented object, and wherein the data indicative of the mass is assigned to each of the labeled segmented objects. Associated The system further comprises a controller configured to perform the steps of:

[0049] The manual process of waste sorting takes a lot of time and resources. Not only is it inefficient, but it is also prone to errors. The present invention provides an improved characterization system that can be employed in sorting systems in some examples. It is possible to characterize and track information of each component, object or particle. The present invention can also be used in other applications, for example in agricultural and / or food processing applications. For example, a batch or stream of strawberries can be characterized in a detailed manner for quality determination. Additionally, based on the characterization, the value of the batch or stream of strawberries can be estimated.

[0050] The invention can also be used in other applications, for example for process control in separation plants (waste recycling plants). In some examples, such plants comprise one or more physical separation units (e.g. sieves, eddy current, magnetic separators, vibration separators, etc.). The effectiveness of the one or more separation units can be evaluated even on the fly and / or in real time. This can allow the device to be optimized and adjusted based on quality control. For example, if the separated particles or parts are larger than a certain value, a certain gap must be adjusted to separate smaller particles or parts. In conventional solutions, this is not possible, because the entire process would be stopped to take samples and perform characterization measurements.

[0051] Advantageously, pre-processing and feature extraction steps can be performed before feeding the data to the trained artificial neural network for labeling the objects. A number of features / characteristics can be determined for each segmented object. A label can then be assigned for each object. The trained neural network can be used for the automatic determination of a label associated with each segmented object, taking into account the determined number of features / characteristics linked to the segmented objects. According to the invention, instead of feeding the imaging data directly to a neural network, the imaging data is segmented in order to identify individual components whose features / characteristics are fed to the trained neural network. Optionally, imaging data from different sensors / detectors are aligned, allowing data fusion. Significantly more accurate results can be obtained in this way. It is also possible to additionally input the imaging data to the trained neural network. Thus, instead of providing the pixels of an image as input, each of the objects is segmented. Association The features / characteristics obtained are given as inputs, which can be done effectively on an object-by-object basis. The inputs to the trained neural network can be, for example, mass, volume, atomic number, shape parameters such as eccentricity, etc. (see input nodes in the first layer of the neural network). The output of the neural network can provide a classification of the segmented object.

[0052] The system can provide efficient and effective characterization of one or more materials (eg, material streams) at relatively high throughput.

[0053] According to one aspect, the present invention provides a computer program product arranged to perform the steps of the method according to the present invention when executed on a controller.

[0054] According to one aspect, the present invention provides a recycling device configured to sort a waste stream, wherein the recycling device comprises a system as defined in claim 12 for distinguishing between different materials in the waste stream.

[0055] Optionally, the recycling device is configured to control process parameters based on information relating to the distinguished different materials in the waste stream.

[0056] The method allows for real-time mass balance, and thus, for example, complete valuation, of well-defined subfractions without the need for separation steps. This allows recycling plants to easily perform chemical analysis in analytical laboratories. Related This could allow for the reduction, and eventually complete skipping, of expensive, labor-intensive sampling procedures, with standard lead times for industrial analysis ranging from several weeks to three months, and reduced uncertainty.

[0057] According to one aspect, the invention relates to the use of the system for the characterization of one or more substances.

[0058] According to one aspect, the invention provides an in-line characterization system adapted for in-line characterization of complex and heterogeneous material streams. The system can be equipped with multiple sensors arranged above a conveyor belt, for example 1 m wide, moving at a speed of up to 1 m / s. The material can be presented in a monolayer on the belt. Optionally, means are provided to ensure the monolayer. The sensors installed include dual-energy x-ray transmission (DE-XRT), which allows to directly measure the average atomic number and mass, and essentially "see through the material", optional 3d laser triangulation (3D-LT), which allows to determine the exact volume and shape, and optionally a high-resolution color camera. The data from these sensors can be processed by a controller, which can be configured to align and fuse the received data of the different sensors / detectors, and to apply models and algorithms based on artificial intelligence and machine learning to detect, classify and measure the individual objects / components on the belt.

[0059] The in-line characterization system can be configured to use X-rays to "see through" materials, not just at the surface layer. This allows for identifying composite materials, e.g. materials whose value is hidden under a protective layer or layer of dust. Furthermore, the integrated dual energy X-ray transmission (DE-XRT) technology allows for the mass of objects to be measured directly, without the need to weigh them directly. The DE-XRT technology also directly measures the average atomic number. Complemented by object volume and 3D shape measurements (using 3D laser triangulation (3D-LT)), density can also be derived. Additionally or alternatively, a high-resolution color camera can be installed, allowing for identifying structures, patterns, colors, shapes, text, etc., using, for example, machine learning models (e.g., deep learning algorithms). Geometric algorithms can be used for precise alignment of sensor data recorded from different viewpoints.

[0060] According to one aspect, the present invention provides a method and a system for characterizing one or more materials, the one or more materials being scanned by a perception system comprising an X-ray sensor configured for multi-energy imaging to obtain at least a lower energy X-ray image and a higher energy X-ray image. Further, a segmentation can be performed on the image acquired by the perception system to separate one or more distinct objects in the image, where data indicative of the area of ​​the segmented objects is determined. Further, for each segmented object, data indicative of area density and data indicative of atomic number can be determined by analysis of the lower energy X-ray image and the higher energy X-ray image, the data indicative of area density and atomic number being determined by a model calibrated by performing multi-energy X-ray imaging with different materials having known area density and atomic number. Further, for each segmented object, the data indicative of at least the atomic number can be provided as an input to a trained neural network, where the trained neural network is configured to label each segmented object.

[0061] It will be appreciated that any of the aspects, features and options described in the context of the method apply equally to the system and the described recycling device and computer program product. It will also be apparent that any one or more of the above aspects, features and options may be combined.

[0062] The invention will be further elucidated on the basis of the exemplary embodiments represented in the drawings, which are given by way of non-limiting illustration, it being noted that the figures are only schematic representations of embodiments of the invention, given by way of non-limiting illustration. [Brief description of the drawings]

[0063] [Figure 1] FIG. 1 shows a schematic diagram of an embodiment of the system. [Diagram 2] FIG. 2 shows a schematic diagram of an embodiment of the system. [Diagram 3] FIG. 3 shows a schematic diagram of the characterized materials. [Figure 4-1] Figure 4-1 shows a schematic diagram of data fusion. [Figure 4-2] Figure 4-2 shows a schematic diagram of data fusion. [Diagram 5] FIG. 5 shows a schematic diagram of the graph. [Figure 6] FIG. 6 shows a schematic diagram of a segmented image. [Figure 7] FIG. 7 shows a schematic diagram of the method. [Figure 8] FIG. 8 shows a schematic diagram of the graph. [Figure 9] FIG. 9 shows a schematic diagram of the graph. [Figure 10A] FIG. 10A shows a schematic diagram of the graph. [Figure 10B] FIG. 10B shows a schematic diagram of the graph. [Figure 11A] FIG. 11A shows a schematic diagram of the graph. [Figure 11B] FIG. 11B shows a schematic diagram of the graph. [Figure 12] FIG. 12 shows a schematic diagram of an embodiment of the system. [Figure 13] FIG. 13 shows a schematic diagram of the method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0064] 1 shows a schematic diagram of an embodiment of a system 1 for characterizing one or more materials 3, the system 1 comprising a perception unit 5 arranged to scan the one or more materials 3. The perception unit 5 comprises an X-ray sensor configured for multi-energy imaging to obtain at least lower energy and higher energy X-ray images. The system 1 includes: segmenting the acquired image by the perception system 5 to separate one or more distinct objects within the image; determining, for each of the segmented objects, data indicative of an area of ​​the segmented objects by analysis of the lower energy X-ray image and the higher energy X-ray image, wherein data indicative of an area of ​​the segmented objects is determined; determining, for each of the segmented objects, data indicative of an area density and data indicative of an atomic number by analysis of the lower energy X-ray image and the higher energy X-ray image, wherein the data indicative of the area density and the atomic number are determined by a model that is calibrated by performing multi-energy X-ray imaging with different materials having known area densities and atomic numbers; calculating, for each of the segmented objects, data indicative of a mass based on the data indicative of the area and the data indicative of the density for each of the segmented objects; and providing, for each of the segmented objects, the data indicative of at least the atomic number as input to a trained neural network, wherein the trained neural network is configured to label each segmented object, wherein the data indicative of the mass is assigned to each of the labeled segmented objects. Associated In this example, the perception unit 5 and the one or more substances 3 are movable relative to each other, as indicated by arrow A. In this way, the perception unit can scan the one or more substances 3. For example, a conveyor belt can be used to guide the one or more substances 3 along the perception unit 5. However, it is also possible that the perception unit 5 is moved relative to the one or more substances 3.

[0065] The present invention allows to assign a mass to each segmented object, which means that a total mass balance can be assigned for each classification of objects.

[0066] In some instances, the mass can be provided as an input to the trained neural network. The combination of mass and chemical properties (see atomic number) provides a good prediction of the label by the trained artificial neural network.

[0067] In some examples, after a classification is assigned to each of the segmented objects using the trained neural network, the mass is used at least in the output, and the mass is used to create the mass balance.

[0068] 2 shows a schematic diagram of an embodiment of the system 1. In this example, at least one of an optional (color) camera 7 or an optional 3D laser triangulation unit 9 is arranged to allow determining additional properties associated with each of the segmented objects. Thus, in some examples, next to features / properties related to material type, mass, etc., it is also possible to make a distinction between the identified and / or segmented objects based on at least one of size, shape, color, texture, visual insight, etc. Such information may also allow virtual experiments. In this example, the perception unit 5 includes an X-ray sensor 11 with two X-ray subunits 11a, 11b for performing dual energy X-ray imaging. Furthermore, the camera 7 and the 3D laser triangulation unit 9 are integrated in the perception unit 5. In this way, the perception unit 5 provides multiple images that can be aligned and / or fused, for example by a computer unit 13. Aligning and / or fusing imaging data acquired from different cameras / detectors may allow for better determination of the features / characteristics of the segmented objects. The one or more materials are segmented and the individual segmented objects 3i are analyzed to determine relevant features / characteristics of the individual segmented objects 3i. In this example, the following features 15 are determined for each segmented object: density, material, shape, size and mass. It will be appreciated that other sets of features are also possible. From the data it is also possible to derive the (relative) weight (proportion) of each of the segmented objects.

[0069] Systems according to the invention can be faster and more autonomous in characterizing one or more materials while requiring less (labor-intensive) input from humans, which can provide important advantages in waste characterization applications.

[0070] To develop a model that recognizes different waste particles and classifies them into different categories, a machine learning model can be trained by showing it many images, each accompanied by a label that describes what is in it. The traditional approach, where all data is pre-labeled, is known as supervised learning. This labeled data represents the fuel for the machine learning algorithm. In the case of waste characterization techniques, the labeled data can typically be generated by scanning physical "pure" single-material streams, which are often prepared manually by selecting thousands of individual particles from heterogeneous waste streams.

[0071] The characterization of waste has several important applications in the recycling industry: Waste characterization can be used for valuation: A fast and reliable valuation of the complete material stream reduces the risk of exposure to commodity stock market volatility. Furthermore, waste characterization can be used for quality control: In a circular economy, it is desired to guarantee the quality of recycled products. The characterization technique helps to establish market confidence. Furthermore, the characterization technique can be used for process engineering: The technical and economic feasibility of waste recycling processes can be evaluated, as well as the design of new processes by virtual experiments. Furthermore, the characterization technique can be used for online process optimization: Sorting processes can be measured, controlled and optimized on the fly.

[0072] The present invention can provide a direct in-line characterization technique to evaluate materials both qualitatively (material type, chemistry, purity, etc.) and quantitatively (mass balance, physical properties, etc.). Such an in-line characterization system can be configured to fully evaluate heterogeneous and complex material streams, eliminating the need for sub-sampling. Furthermore, mass balances can be generated on-the-fly. Indeed, for each material object, a digital twin can be created that can be further evaluated in a virtual manner.

[0073] Fig. 3 shows a schematic diagram of a material characterized using the method according to the invention. Different objects 3i are segmented and relevant properties / features are determined. The information can be presented, for example, using a generic user interface. In this example, the features include mass, volume, diameter, shape and texture. Furthermore, a label 17 determined by the trained neural network can also be provided. In this example, two labels 17 are provided: object type (e.g. bottle, camera, headphones, etc.) and material (e.g. PET, material composition, etc.). It will be appreciated that other features and / or labels are also possible.

[0074] 4 shows a schematic diagram of data fusion. In this example, the perception unit 5 includes multiple imaging sensors, namely an RGB color camera, a height sensor (e.g., a laser triangulation or 3D camera), and an X-ray sensor (e.g., a dual energy X-ray sensor providing a low energy X-ray image and a high energy X-ray image). It will be appreciated that instead of using two separate sensors, it is possible to integrate two separate sensors. For example, a 3D camera can be used to provide both height information and an RGB color image. The fused image 19 can be the acquired images from different sensors or sub-units of the imaging unit 5.

[0075] The absorption rate of X-rays is measured by X-ray imaging. The absorption rate of X-rays can be proportional to the mass of a particular material. However, a relatively thin material with high absorption rate (e.g. lead) can give a similar X-ray image as a very thick material with low absorption rate. Lower and higher energy images can be used to distinguish between such cases, allowing the differentiation of materials.

[0076] By combining multiple sensors (e.g., XRT and 3DLT), the physical / chemical properties of the segmented objects can be determined or directly measured. The measured and / or determined properties can be provided as input to machine learning models to provide labels. The value and composition of complex and heterogeneous material streams can be quantified, resulting in cost and time savings for recycling companies compared to current sampling and analysis procedures.

[0077] Figure 5 shows a schematic diagram of a graph in which actual mass is plotted against predicted mass for different material types. It can be seen that the model for determining the mass based on lower and higher energy X-ray images provides accurate results. The mass relative error is less than 15 percent.

[0078] Fig. 6 shows a schematic diagram of a segmented image. The segmentation of the objects 3i is visualized by a bounding box 20, which in this example is rectangular, but other shapes are also possible. Other segmentation techniques are also possible. For example, it is also possible to segment the contours of the identified objects 3i. For each segmented object 3i, one or more features / characteristics can be determined. These characteristics can be fed to the trained neural network to provide application-specific labels.

[0079] FIG. 7 shows a schematic diagram of the method 30. In a first step 31, the object or component of the one or more substances is identified and segmented. This can be done by an object detection algorithm and / or a segmentation algorithm. Images are acquired using the perception unit 5. The acquired images that are segmented can also be acquired after alignment and / or fusion of different images, e.g. coming from different sensors or sub-units of the perception unit 5. In this example, a box 20 is provided around the segmented object 3i. In a second step 33, properties / characteristics 15 are determined for each segmented object 3i. In this example, mass, volume and atomic number are determined. Data can be provided as input to the trained neural network 25 to obtain a label 17 as output. In this example, the trained neural network is a deep learning model. However, other machine learning models can also be used. In some examples, instead of an artificial neural network, an alternative regression model is used.

[0080] Figure 8 shows a schematic diagram of a graph. The plurality of identified segmented objects 3i can be divided into one or more clusters, such that each cluster contains components with similar features and / or characteristics. The objects can be grouped in this way to provide more insight into the composition of the one or more substances (e.g. waste streams). In this example, 2D clusters are provided that take into account density and atomic number. It will be appreciated that various other cluster graphs are possible.

[0081] The present invention allows for material stream characterization at object / component level. In addition to particle size distribution or mass balance, custom quality metrics can be defined based on (directly measured) physical properties of individual objects of the one or more materials (e.g., material streams). In doing so, the present technology builds a digital twin of the physical material stream, enabling virtual experiments, new insights and better valuation. The present invention allows for sorting of heterogeneous waste streams with high classification accuracy and accurate mass balance without the need for a sorting step.

[0082] Figure 9 shows a schematic diagram of a graph similar to that of Figure 8. In this example, the identified segmented objects 3i are shown in the graph, allowing for more visual insight into the composition of the one or more materials being characterized.

[0083] 10A and 10B show schematic diagrams of graphs relating to dual-energy and multi-energy X-ray transmission, in which the transmission is plotted against energy for different segmented objects 3i.

[0084] When (X-ray) photons pass through a material, some of them interact with the material while another part does not. This last part, the transmitted part, is what is captured by the detector in X-ray transmission (XRT) imaging. The object to be imaged is located between the source and the detector.

[0085] The amount of x-rays transmitted by a material depends on 1) the material (chemical composition, atomic number), 2) density (ρ), 3) thickness (d), and 4) the energy of the photons (E).

[0086] This is reflected in Figure 10A above, where photon transmission is plotted against X-ray photon energy for different materials. Due to the different spectral transmittance of materials, we are able to distinguish between different materials when imaging them using multiple energies.

[0087] For example, in the case of dual energy XRT (DE-XRT), each sensor pixel will sense in two parts of the energy spectrum and therefore generate a low energy signal and a high energy signal. The low energy signal integrates all photons below a certain threshold, while the high energy signal integrates X-rays and energies above the threshold. By measuring the transmission in the low and high parts of the energy spectrum, it is possible to calculate the average atomic number (Z) and areal density (ρd) (density x thickness).

[0088] As can be seen, the transmission spectrum of a material shows individual dips (the transmission spectrum of a material also has peaks, but these are not depicted). The location of these dips (at the X-ray photon energy) is characteristic for a chemical element, while the height of the dips corresponds to the amount of the material. Therefore, by introducing multiple energy bins (more than two) and performing multi-energy X-ray transmissions, it is possible to focus on some or all of these dips, and therefore, by examining the location and the height of the dips, it is possible to quantitatively determine the chemical (elemental) composition of the scanned material.

[0089] Dual energy is capable of measuring average atomic numbers within a certain range (e.g., up to atomic number 40). However, multi-energy transmission can greatly extend this range to higher atomic numbers. Also, by focusing on specific drops, multi-energy XRT allows one to focus on specific chemical elements and quantitatively determine the amount of that element present in a material. One known technique for this is K-edge imaging.

[0090] In Figure 10B, the effect of areal density (ρd) (density x thickness) is depicted. The higher the density or the thicker the material, the lower the transmittance. However, the location of the dip remains the same. The relative height of the dip is also very similar.

[0091] 11A and 11B show schematic diagrams of graphs relating to dual-energy and multi-energy X-ray transmission. In Fig. 11A, low-energy and high-energy images are shown. In Fig. 11B, the high-energy transmission is plotted against the low-energy transmission for different segmented objects 3i.

[0092] In FIG. 11A, the skilled person can see the effect of increasing Pb content. The position of the dip remains constant, but the height of the dip increases with increasing Pb content. This effect allows the chemical composition of a material to be quantitatively determined using multi-energy imaging, e.g. k-edge imaging. Based on the DE-XRT calibration scan, the following relationship between LE, HE and Z can be derived. This allows the average atomic number to be derived based on the measurements of LE and HE, as illustrated in FIG. 11B. Similarly, the relationship with ρd can be derived.

[0093] During a calibration period, a material can be sampled whose atomic number and density are known. The graph shows what the atomic number and surface density are for the low energy x-ray image and the high energy x-ray image. When the surface density is integrated or added to the segmented object, the mass of the object can be calculated.

[0094] 12 shows a schematic diagram of an embodiment of a system 100. The system 100 can be configured to control process parameters based on information related to the differentiated different materials in the waste stream. The system 100 can be configured to provide real-time process information for process control. For example, in a process, after a characterization step, there is a boundary that includes, for example, multiple different physical separation steps. Based on what inputs are coming in, the parameters (e.g., speed bands, machine settings) can be adjusted based on what is presented as input to achieve better separation.

[0095] The present invention can provide a direct in-line characterization technique for bulk solid waste streams. Characterization utilizes different sensors (X-ray, 3D laser and color) and artificial intelligence to provide industrially relevant analytics. Related The method may convey object / component level big data of the one or more substances (e.g., substance streams) that have been collected.

[0096] FIG. 13 shows a schematic diagram of a method 1000 for characterizing one or more materials. In a first step 1001, the one or more materials are scanned by a perception system comprising an X-ray sensor configured for multi-energy imaging to obtain at least a lower energy X-ray image and a higher energy X-ray image. In a second step 1002, a segmentation is performed on the image acquired by the perception system to separate one or more distinct objects in the image, and data indicative of the area of ​​the segmented objects is determined. In a third step 1003, data indicative of area density and data indicative of atomic number are determined for each segmented object by analysis of the lower energy X-ray image and the higher energy X-ray image, the data indicative of area density and atomic number being determined by a model calibrated by performing multi-energy X-ray imaging with different materials having known area density and atomic number. In a fourth step 1004, data indicative of mass for each segmented object is calculated based on the data indicative of area density and the data indicative of area for each segmented object. In a fifth step 1005, for each of the segmented objects, the data indicative of at least the atomic number is provided as an input to a trained neural network, where the trained neural network is configured to label each segmented object, where the data indicative of the mass is assigned to each of the labeled segmented objects. Associated do.

[0097] Knowledge of the mass using an X-ray imaging unit can provide significant advantages in material characterization, which can even be performed on complex and heterogeneous streams (e.g., applications for sorting waste material streams).

[0098] Various neural network models and / or neural network architectures can be used. Neural networks have the ability to process, e.g. classify, sensor data and / or pre-processed data (see determined feature properties of the segmented object). Neural networks can be implemented in computerized systems. Neural networks can serve as a framework for various machine learning algorithms to process complex data inputs. Such neural network systems can generally "learn" to perform a task by considering examples, without any task-specific rules also being programmed. Neural networks can be based on a collection of connected units or nodes, called neurons. Each connection can send a signal from one neuron to another in the neural network. A neuron that receives a signal can process the signal and then send the signal to additional neurons connected to it (see activation). The output of each neuron is typically calculated by some nonlinear function of the sum of its inputs. The connections can have respective weights that are adjusted as learning progresses. There can also be other parameters, e.g. biases. Typically, the neurons are aggregated into layers. Different layers may perform different types of transformations on their inputs to form a deep neural network.

[0099] Deep learning neural networks can be considered as a representation learning method with multiple levels of representation, which can be obtained by constructing simple but nonlinear modules, each of which converts a representation at one level, starting from raw input, into a representation at a higher, slightly more abstract level. The neural network can identify patterns that are difficult to see using conventional or classical methods. Thus, instead of writing custom code specific to the problem of printing structures in certain printing conditions, the network can be trained to handle different and / or changing structure printing conditions, for example using classification algorithms. Training data can be provided to the neural network so that it can determine classification logic to efficiently control the printing process.

[0100] When a particular step of a method is referred to as following another step, it will be further understood that the particular step may directly follow the other step or, unless otherwise specified, one or more intermediate steps may be performed prior to performing the particular step. Similarly, when a connection between components, such as neurons of the neural network, is described, it will be understood that the connection may be established directly or through intermediate components, such as other neurons or logical operations, unless otherwise specified or excluded by context.

[0101] It will be appreciated that the method may include computer-implemented steps. All steps mentioned above may be computer-implemented steps. The embodiment may comprise a computing device, the method being performed in a computing device. The invention also extends to a computer program, in particular a computer program on or in a carrier adapted for carrying out the invention. The program may be source code or object code or any other form suitable for use in implementing the method according to the invention. The carrier may be any entity or device capable of carrying the program. For example, the carrier may comprise a storage medium, such as a ROM, for example a semiconductor ROM or a hard disk. Furthermore, the carrier may be a transmissible carrier, such as an electrical or optical signal that may be conveyed via an electrical or optical cable, or by radio or other means, for example via the Internet or the cloud.

[0102] Some embodiments may be implemented, for example, using a machine or tangible computer-readable medium or article that may store instructions or sets of instructions that, when executed by a machine, cause the machine to perform methods and / or operations in accordance with the embodiments.

[0103] Various embodiments may be implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements may include processors, microprocessors, circuits, application specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), logic gates, registers, semiconductor devices, microchips, chipsets, etc. Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, mobile apps, middleware, firmware, software modules, routines, subroutines, functions, computer-implemented methods, procedures, software interfaces, application program interfaces (APIs), methods, instruction sets, computing code, computer code, etc.

[0104] Graphics and / or image / video processing techniques may be implemented in a variety of hardware architectures. Graphics functionality may be integrated within a chipset. Alternatively, a separate graphics processor may be used. For example, processing of images (still or video) may be performed by a graphics subsystem, such as a graphics processing unit (GPU) or a visual processing unit (VPU). In yet another embodiment, the graphics or image / video processing functions may be implemented by a general-purpose processor, including, for example, a multi-core processor. In a further embodiment, the functions may be implemented in a consumer electronics device. Implementations using a combination of different hardware architectures are possible.

[0105] The present invention has been described herein with reference to specific examples of embodiments of the present invention. However, it will be apparent that various modifications, variations, substitutions and changes can be made without departing from the essence of the present invention. For the purpose of clarity and concise description, features are described herein as part of the same or different embodiments, but it is anticipated and understood that alternative embodiments having all or part of the features described in these different embodiments also fall within the framework of the present invention as outlined by the claims. Thus, the specification, figures and examples should be considered in an illustrative sense, not a restrictive sense. The present invention is intended to cover all alternatives, modifications and variations that fall within the spirit and scope of the appended claims. Moreover, many of the described elements are functional entities that can be implemented as separate or distributed components, or with other components in any suitable combination and location.

[0106] In the claims, any reference signs placed between parentheses shall not be interpreted as limiting the claim. The word "comprising" does not exclude the presence of other features or steps than those recited in a claim. Moreover, the words "a" and "an" shall not be interpreted as being limited to "only one" but are instead used to mean "at least one" and do not exclude a plurality. The mere fact that certain measures are recited in mutually different claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. 1. A computer-implemented method for characterizing one or more substances, comprising: scanning the one or more materials with a perception system including an x-ray sensor configured for multi-energy imaging to obtain at least a lower energy x-ray image and a higher energy x-ray image; segmenting the acquired image by the perception system to isolate one or more distinct objects within the image, where data indicative of areas of the segmented objects is determined; determining, for each of the segmented objects, data indicative of an area density and data indicative of an atomic number by analysis of the lower energy x-ray image and the higher energy x-ray image, where the data indicative of the area density and the atomic number are determined by a model that is calibrated by performing multi-energy x-ray imaging with different materials having known area densities and atomic numbers; calculating, for each of the segmented objects, data indicative of a mass based on the data indicative of the areal density and the data indicative of the area for each of the segmented objects; and providing, for each of the segmented objects, the data indicative of at least the atomic number as an input to a trained neural network, wherein the trained neural network is configured to label each segmented object, wherein the data indicative of the mass is associated with each of the labeled segmented objects; The method comprising:

2. The method of claim 1 , wherein the calculated mass is provided as an input to the trained neural network.

3. The method of claim 1 or 2, wherein the perception system further comprises a depth imaging unit for determining data indicative of a volume of the segmented object.

4. The method of claim 3 , wherein the depth imaging unit comprises at least one of a 3D laser triangulation unit or a 3D camera.

5. The method according to any one of claims 1 to 4, wherein the perception system further comprises a colour imaging unit configured to capture colour images of the segmented object.

6. A method according to any one of claims 1 to 5, wherein data from different subsystems of the perception system are aligned prior to determining characteristic features for each of the segmented one or more objects.

7. The method according to any one of claims 1 to 6, wherein for each of the segmented objects further characteristic features related to at least one of volume, size, diameter, shape, texture, colour or eccentricity are determined.

8. 8. A method according to any one of claims 1 to 7, wherein the one or more materials are moved on a conveyor forming a material stream, wherein the material stream is scanned by the perception system for characterisation of objects within the material stream.

9. A method according to any one of claims 1 to 8, wherein characteristic features of the segmented one or more objects are stored to construct a digital twin model.

10. 10. The method of claim 1, wherein the one or more substances are characterized prior to transportation to determine a first digital identification marker, and wherein subsequently after transportation to a remote location the one or more substances are characterized to determine a second digital identification marker, and wherein the first digital identification marker and the second digital identification marker are compared to each other to determine a change in content during the transportation period.

11. The method of any one of claims 1 to 10, wherein the one or more materials are heterogeneous.

12. The method of any one of claims 1 to 11, wherein the one or more materials are selected from the group consisting of solid waste, manufactured products or components, agricultural products, or batteries.

13. 1. A system for characterizing one or more substances, comprising: a perception unit arranged to scan the one or more materials, where the perception unit comprises an X-ray sensor configured for multi-energy imaging to obtain at least a lower energy X-ray image and a higher energy X-ray image; Here, the system segmenting the image acquired by the system to isolate one or more distinct objects within the image, where data indicative of areas of the segmented objects is determined; determining, for each of the segmented objects, data indicative of an area density and data indicative of an atomic number by analysis of the lower energy x-ray image and the higher energy x-ray image, where the data indicative of the area density and the atomic number are determined by a model that is calibrated by performing multi-energy x-ray imaging with different materials having known area densities and atomic numbers; calculating, for each of the segmented objects, data indicative of mass based on the data indicative of the areal density and the data indicative of the area for each of the segmented objects; providing, for each of the segmented objects, the data indicative of at least the atomic number as an input to a trained neural network, wherein the trained neural network is configured to label each segmented object, wherein the data indicative of the mass is associated with each of the labeled segmented objects. The above system further comprises a controller configured to perform the steps of:

14. A computer program product comprising one or more computer readable storage devices having program instructions stored thereon for performing each of the steps of the method according to any one of claims 1 to 12, when executed on a controller.

15. 14. A recycling device configured to sort a waste stream, wherein the recycling device comprises the system of claim 13 to distinguish between different materials in the waste stream.

16. 16. The recycling device of claim 15, wherein the recycling device is configured to control process parameters based on information related to the distinguished different materials in the waste stream.

Images