Neural Networks for Bulk Sorting

Abstract

A bulk sorting system for sorting objects (1) in bulk comprises at least one radiation source (10) arranged to radiate radiation onto the objects, at least one optical sensor (12) arranged to capture reflected radiation (22) from the objects and obtain the reflected radiation as multispectral or hyperspectral data (24), a processing circuit (16) configured to analyze the reflected radiation from the objects by inputting the multispectral or hyperspectral data into a convolutional neural network (CNN) (18) having at least two convolutional layers to detect and classify the objects in the multispectral or hyperspectral data and / or to semantically segment the multispectral or hyperspectral data, and a mechanical sorter (20) configured to sort the objects according to their classification and / or segmentation using the analysis of the processing circuit, such that different overlapping and / or stacked objects are separated or treated as a single group of objects.

Description

[Technical field]

[0001] The present invention relates to the field of sorting, and more specifically to bulk sorting assisted by convolutional neural networks (CNNs). [Background technology]

[0002] Sorting is a topical research field that affects, for example, recycling, mining or food processing. In recycling implementations, sorting techniques are used to separate mixtures of garbage into the correct bins. As technology evolves, this sorting can be performed more accurately and faster than before.

[0003] Techniques exist for using machine learning systems to sort streams of single objects, such as those shown in US2018 / 243800AA. The machine learning system can accurately identify the objects being sorted. However, such techniques are slow, as they can only process one object stream at a time. Other examples of material characterization and segmentation techniques can be found in US2019 / 130560A1 and Matthieu Grardet al:”Object segmentation in depth maps with one user click and a synthetically trained fully convolutional network”,2018.

[0004] Current sorters separate individual particles. They require careful feed preparation so that individual particles are detected and measured, and discharge is usually achieved by a blast of compressed air. The capacity of current sorters is therefore very low (up to 300 tonnes per hour for larger particles, much less for smaller ones) and they are not feasible for preconcentration or so-called bulk sorting at higher tonnages. The sorting speed and throughput achieved in bulk sorting are directly related to the size of the objects being sorted. In general, sorting speed and throughput vary widely for large and small items and depend heavily on the specific application, which can range from food, paper, wood, plastic and mineral sorting applications.

[0005] For sorting to be feasible, all preconcentrations must be applied to large volumes of objects, such as loaded truck trays or fully loaded conveyor belts.

[0006] Therefore, improvements are needed in this context. Summary of the Invention

[0007] The present invention therefore seeks to solve at least some of the above problems and to eliminate or at least mitigate some of the drawbacks of prior art systems. This object has been achieved according to the present invention by novel techniques as set out in the accompanying independent claims. Preferred embodiments are defined in the associated dependent claims.

[0008] It is a further object of the present invention to provide a sorting system capable of sorting a plurality of objects in bulk. According to a first aspect, the above and other objects of the present invention are achieved in whole or in part by a bulk sorting system for sorting objects in bulk, comprising at least one radiation source arranged to radiate onto the objects, at least one optical sensor arranged to capture reflected radiation from the objects and obtain the reflected radiation as multi- or hyperspectral data, a processing circuit configured to analyze the reflected radiation from the objects by inputting the multi- or hyperspectral data into a convolutional neural network (CNN) (18) having at least two convolutional layers in order to detect and classify the objects in the multi- or hyperspectral data and / or to semantically segment the multi- or hyperspectral data, and a mechanical sorter configured to sort the objects using the analysis of the processing circuit.

[0009] This is beneficial in that previous issues such as objects overlapping each other and interfering with the sensor data can be identified and processed accordingly. A CNN with at least two convolutional layers can be advantageously trained to process this type of multispectral or hyperspectral data in an improved manner compared to known sorting systems for bulk sorting. Object processing can be performed using overlapping objects through detection and classification and / or semantic segmentation using a CNN. This allows for a higher sorter throughput per hour compared to traditional processing methods. In one embodiment, the at least one optical sensor includes a near-infrared scanner positioned to scan the object, and the multispectral or hyperspectral data includes the scan data.

[0010] NIR scanners are beneficial in that they can extract a lot of characteristic information from the NIR absorption of the measured object.

[0011] In one embodiment, the at least one optical sensor includes an image sensor positioned to capture image data of the object, and the multispectral or hyperspectral data includes the image data.

[0012] Image sensors are beneficial in that many image processing algorithms exist for image data, such as object recognition, image segmentation, etc. Furthermore, CNNs are typically well suited to analyzing image data.

[0013] In one embodiment, the at least one optical sensor includes a hyperspectral camera positioned to scan the object, and the multispectral or hyperspectral data includes the scan data.

[0014] Hyperspectral cameras are beneficial in that they are an efficient way of collecting large amounts of data for use in CNNs.

[0015] In one embodiment, the at least one optical sensor includes a laser triangulator arranged to measure 3D data of the object, and the multispectral or hyperspectral data includes the 3D data, and the measuring of the 3D data may include laser height intensity scanning. Laser triangulation is beneficial in that it allows for precise 3D measurements, which are beneficial for CNN analysis.

[0016] In one embodiment, the system further includes an electromagnetic detector positioned to measure electromagnetic properties of the object, and the processing circuitry is further configured to analyze the electromagnetic properties by inputting the measured electromagnetic properties into a CNN to detect and classify objects in the multispectral or hyperspectral data and / or to semantically segment the multispectral or hyperspectral data.

[0017] Electromagnetic detectors are beneficial in that classification of certain types of objects, such as metals and isolators, is much more accurate using their electromagnetic properties.

[0018] In one embodiment, the at least one optical sensor includes a laser scanner with a rotating polygon mirror arranged to measure laser scattering and / or anti-scattering properties of the object, and the multispectral or hyperspectral data includes the laser scattering and / or anti-scattering properties.

[0019] Laser scanners are beneficial in that a lot of characteristic information can be extracted from the measured laser scattering and / or anti-scattering properties of objects. The rotating polygon mirror allows the measurement of multiple objects and their scattering properties in different orientations without the need for a target.

[0020] In one embodiment, the at least one optical sensor comprises a pulsed LED emitter arranged to measure anti-light scattering properties of the object, and the multi-spectral or hyper-spectral data includes the anti-light scattering properties.

[0021] LED emitters are beneficial in that they require little maintenance and calibration and generate relatively little heat.A lot of distinctive information can be extracted from the measured anti-light-scattering properties of an object.

[0022] In one embodiment, the at least one optical sensor includes an X-ray camera positioned to measure X-ray transmission through the object, and the multi-spectral or hyper-spectral data, respectively, includes X-ray transmission through the object.

[0023] X-ray cameras are useful in that they can measure many different properties at once. The dielectric constant of objects to X-ray radiation may indicate their atomic density and / or their thickness. X-ray fluorescence results are information about the presence of elements and their concentrations.

[0024] In one embodiment, the system further includes using a conveyor belt to transport the objects along a detection range of the at least one optical sensor to a mechanical sorter.

[0025] Conveyor belts are beneficial in that they allow for steady, fast and predictable movement of objects. An alternative to conveyor belts is a chute.

[0026] In one embodiment, the mechanical sorter is further configured to separate the objects into at least two streams and / or to eject unwanted objects from the bulk.

[0027] Separation of objects is beneficial in that several sets of objects may be needed from the bulk. Discharge of unwanted objects is beneficial in that there are frequently unwanted objects in the bulk that will interfere with future processes.

[0028] In one embodiment, the mechanical sorter comprises at least one valve positioned to discharge at least one air stream that pushes the objects to a desired location.

[0029] The at least one valve is beneficial in that it is an energy efficient and precise method for pushing an object. The valve is preferably a solenoid valve as it has a relatively short switching time and a relatively high throughput performance.

[0030] In one embodiment, the mechanical sorter includes at least one mechanical kicker positioned to kick objects to a desired location.

[0031] At least one mechanical kicker is beneficial in that it is an energy efficient and powerful way to push an object. Preferably, the kicker is pneumatic.

[0032] In one embodiment, the mechanical sorter is arranged to sort objects as they move along a conveyor belt, as they move along a chute, or as they free fall.

[0033] Sorting on the move is beneficial in that it is efficient in terms of speed. The sorter may be located near the conveyor belt and positioned to move objects in a direction perpendicular to the direction of motion of the conveyor belt. The chute or free fall may be located at or along the edge of the conveyor belt. In this embodiment, it is a priority that the mechanical sorter be able to react quickly, since the speed of the objects is relatively difficult to predict and control. In one embodiment, the mechanical sorter has an actuation accuracy of less than 1 millisecond for small particles and less than 10 milliseconds for larger particles such as bottles.

[0034] In one embodiment, the mechanical sorter is configured to sort the objects such that different overlapping and / or stacked objects are separated, as analyzed by the processing circuitry.

[0035] Separation of overlapping and / or stacked objects is beneficial in that different objects may be processed differently despite being overlapping and / or stacked, which increases sorting accuracy and yield.

[0036] In one embodiment, the mechanical sorter is arranged to sort the objects such that different overlapping and / or stacked objects are treated as a single group of objects as analyzed by the processing circuitry.

[0037] Grouping overlapping and / or stacked objects is beneficial in that they may be difficult to separate, especially with certain mechanical sorters.

[0038] In one embodiment, overlapping or stacked objects are treated as a single group of objects, and for a group of objects including at least a first object type and a second object type, the mechanical sorter is configured to sort the group of objects as either the first object type or the second object type based on a configuration of the bulk sort system.

[0039] Sorting the groups as a single object type is beneficial in that in some embodiments it is more valuable to prevent needed material from being discharged or to prevent unneeded material from being retained: in the former case the group is sorted as a needed object type, and in the latter case the group is sorted as an unneeded object type.

[0040] In one embodiment, the mechanical sorter is positioned to target the center of gravity or boundary of the object to be analyzed by the processing circuitry.

[0041] Targeted mechanical sorting is beneficial in that it allows the mechanical sorter to be more efficient in its execution. It uses fewer valves and kickers, and objects move without rotating. This improves accuracy and reduces energy consumption. For example, one-third of the mechanical sorter could be used for the same result if it was better targeted.

[0042] Sorting of objects requires, depending on the size of the objects to be sorted, speeds of 0.4-20 m / s, throughputs of 0.5-30 tonnes / hour and accuracy levels of 80% or higher, preferably 90% or higher with a throughput of a single stage, higher than 95%, and preferably higher than 99% with a cascade of sorting systems which can be achieved with the sorting system and method of the present application.

[0043] In one embodiment, the CNN includes at least two pooling layers. Pooling layers are beneficial in that they downsample the data, reducing the size of the neural network and making the CNN more efficient. A CNN can include any number of pooling layers, including 3, 5, 10, 20, 100, 250, etc.

[0044] In one embodiment, the system further includes post-processing the classified and / or segmented multispectral or hyperspectral data to set up a mechanical sorter before sorting occurs.

[0045] Post-processing is beneficial in that the output can be transformed into something that is more easily interpreted by a mechanical sorter, which may include configuration instructions for the mechanical sorter that are created before sorting of objects corresponding to the classified and / or segmented multispectral or hyperspectral data occurs.

[0046] In one embodiment, the processing circuitry is further configured to input at least a portion of the multispectral or hyperspectral data to a pattern recognition algorithm, where results of both the CNN and the pattern recognition algorithm are used by a mechanical sorter to sort the objects.

[0047] Pattern recognition is beneficial in that it is well established and has the potential to improve the results of CNNs. It also requires no training and can be completed relatively quickly, which can be used when CNNs are not available or are too slow. Such hybrid computation has unexpected synergies, as different analytical methods can yield different information.

[0048] According to a second aspect, the above and other objects of the present invention are achieved in whole or in part by a method for collectively classifying a plurality of objects, the method comprising the steps of irradiating the objects with at least one radiation source, capturing reflected radiation of the objects with at least one optical sensor, acquiring the reflected radiation as the multispectral or hyperspectral data, analyzing the reflected radiation of the objects by inputting the multispectral or hyperspectral data into a Convolutional Neural Network (CNN) having at least two convolutional layers in order to detect and classify the objects in the multispectral or hyperspectral data and / or to semantically segment the multispectral or hyperspectral data, and sorting the objects using the results of the analyzing step with a mechanical sorter.

[0049] This is beneficial in that previous issues such as objects overlapping each other and interfering with the sensor data can be identified and dealt with accordingly. Object processing can be performed with overlapping objects through detection and classification using CNNs and / or semantic segmentation. This results in a higher sorter throughput per hour compared to traditional processing methods. Sorter throughput per hour can be increased by at least 50% compared to traditional processing methods. Object sorting can achieve speeds of 0.4-20 m / s and throughputs of 0.5-30 tons / hour, depending on the size of the objects to be sorted.

[0050] In one embodiment, the method further comprises the step of post-processing the classified and / or segmented multispectral or hyperspectral data to configure a mechanical sorter prior to the sorting step.

[0051] A post-processing step is beneficial in that the output can be transformed into something that is more easily interpreted by a mechanical sorter. This may include setup instructions for the mechanical sorter that are created prior to the sorting step of the objects that correspond to the classified and / or segmented multispectral or hyperspectral data.

[0052] Other objects, features and advantages of the present invention will become apparent from the following detailed disclosure, the appended claims, and the drawings. It is noted that the present invention relates to all possible combinations of features.

[0053] It should be emphasized that the term "comprises / comprising" as used in this specification is used to specify the presence of a stated feature, integer, step, or component, but does not exclude the presence or addition of one or more other features, integers, steps, components, or groups thereof. All terms used in the claims are to be interpreted according to their ordinary meaning in the art, unless otherwise specified in this specification. Unless otherwise stated, all references to "a / an / the [element, device, component, means, step, etc.]" are openly interpreted as referring to at least one instance of the element, device, component, means, step, etc. [Brief description of the drawings]

[0054] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Figure 1] FIG. 1 illustrates a bulk sorting system according to one embodiment. [Diagram 2] FIG. 2 illustrates a bulk sorting system according to one embodiment. [Figure 3a] FIG. 3a shows the result of object detection according to one embodiment. [Figure 3b] FIG. 3b shows the result of semantic segmentation according to one embodiment. [Figure 4] FIG. 4 illustrates a mechanical sorter according to one embodiment. [Figure 5a] FIG. 5a illustrates a bulk sorting system including a near-infrared scanner and an image sensor, according to one embodiment. [Figure 5b] FIG. 5b shows a bulk sorting system including a laser triangulation device and an electromagnetic detector, according to one embodiment. [Figure 6a] FIG. 6a shows a bulk sorting system including a laser scanner, a pulsed LED emitter, and an X-ray camera, according to one embodiment. [Figure 6b] FIG. 6b illustrates a bulk sorting system including a hyperspectral camera according to one embodiment. [Figure 7] FIG. 7 shows a flow chart of the operation of the processing circuit according to one embodiment. [Figure 8] FIG. 8 shows a flow chart of the hybrid operation of the processing circuitry according to one embodiment. [Figure 9] FIG. 9 illustrates a flow chart of a hybrid operation of a processing circuit using two computing units according to one embodiment. [Figure 10] FIG. 10 illustrates a flowchart of a method for sorting objects into chunks, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0055] Next, an embodiment of the present invention will be described with reference to the accompanying drawings. However, the present invention can be embodied in many different forms and should not be construed as being limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. The terms used in the detailed description of the specific embodiments shown in the accompanying drawings are not intended to be limiting of the present invention. In the drawings, like numerals refer to like elements.

[0056] First, in Figs. 1 and 2, a bulk sorting system is shown, and Fig. 2 shows a more detailed example of the same bulk sorting system as in Fig. 1. The system is set up to sort a number of objects 1 in bulk. The bulk objects 1 may include a relatively large number of objects 1 that are randomly spatially separated. The bulk objects 1 are traditionally not a single stream, but an unsorted set that includes potentially overlapping objects 1. Bulk sorting traditionally aims to separate a specific object or objects 1 from other objects 1, i.e., some objects are needed and others are not. An example would be separating plastic from other types of trash so that the plastic can be recycled.

[0057] The system comprises at least one radiation source 10 arranged to radiate onto the object 1. The radiation source 10 may be an LED, a bulb, a laser, an X-ray emitter and / or any other suitable radiation source. The radiation source 10 may be arranged as different units for ambient lighting and / or to radiate in different directions.

[0058] The system further comprises at least one optical sensor 12 arranged to capture reflected radiation 22 of the object 1 and acquire the reflected radiation 22 as multispectral or hyperspectral data 24. The multispectral data 24 may be a multidimensional image where each pixel or equivalent (such as a superpixel or grouping of pixels) comprises several bands (on the order of 1 to 10) with narrow or wide spectra (on the order of 100 to 1000 nm). The hyperspectral data 24 may be a multidimensional image where each pixel or equivalent (such as a superpixel or grouping of pixels) comprises many bands (on the order of 4 to 500) with narrow spectra (on the order of 1 to 50 nm).

[0059] The system further comprises a processing circuit 16 configured to analyze the reflected radiation 22 of the object 1 by inputting the multispectral or hyperspectral data 24 into a convolutional neural network (CNN) 18 having at least two convolutional layers to detect and classify the object 1 in the multispectral or hyperspectral data 24 and / or to semantically segment the object 1 within the multispectral or hyperspectral data 24. More than two convolutional layers, such as 3, 5, 10, 20, 100, 250, etc., can be used.

[0060] The processing circuitry 16 is configured to perform operations and functions of the bulk sorting system. An operation may be a main procedure of the bulk sorting system, whereas a function may form part of an operation. Thus, each function may be a sub-procedure of an operation.

[0061] The processing circuitry 16 may include a processor such as a central processing unit (CPU), a microcontroller, a microprocessor, a field-programmable gate array (FPGA), a graphics card, or special hardware for CNN. The processing circuitry 16 is configured to execute program code stored in memory to perform the operations and functions of the bulk sorting system.

[0062] The operations and functions of the bulk sorting system may be embodied in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored in the bulk sorting system's non-transitory computer-readable medium (e.g., memory) and executed by processing circuitry 16 (e.g., using a processor). Furthermore, the operations and functions of the bulk sorting system may be standalone software applications or may form part of a software application that performs additional tasks related to the bulk sorting system. The described operations and functions may be considered as methods that a corresponding device is configured to perform. Also, while the described operations and functions may be implemented in software, such functions may be performed via dedicated hardware or firmware, or some combination of hardware, firmware and / or software.

[0063] The CNN 18 is a neural network with at least two convolutional layers. The neural network 18 has an input of multispectral or hyperspectral data 24 and an output of multispectral or hyperspectral data 24 in which objects are detected and classified and / or semantically segmented. Examples of the output may be shown in Figures 3a-b. CNN18 may further include at least two pooling layers, which reduce the size of the neural network by downsampling the data, making CNN18 more efficient.

[0064] Detecting and classifying objects in the multispectral or hyperspectral data 24 involves using a CNN 18 (optionally in combination with preprocessing, see below) to classify different portions of the multispectral or hyperspectral data 24 as different objects 1. The objects 1 may be further analyzed and classified into different sets, such as based on the material the objects 1 are most likely made of. The different sets may be further identified as wanted or unwanted to allow for bulk sorting, such as the rejection of unwanted objects 1.

[0065] Semantically segmenting objects in the multispectral or hyperspectral data 24 includes using a CNN to classify each pixel of the multispectral or hyperspectral data 24. Classification may include separation into different sets, such as based on the material the pixel is most likely to be made of. Different sets can be further identified as wanted or unwanted (by the CNN 18 or in combination with preprocessing, see below) to enable bulk sorting, such as the rejection of unwanted objects.

[0066] Figure 3a shows multispectral or hyperspectral data with detected and classified objects. Detections are shown as boxes around the objects and classifications are shown as patterns on the objects. The detected objects are classified into two sets: wanted and unwanted. The unwanted set is shown as a line pattern with dashed boxes and the wanted set is shown as a dotted pattern with solid boxes.

[0067] FIG. 3b shows multispectral or hyperspectral data with semantically segmented objects. Each pixel or equivalent is classified into a set based on the object material, for example according to a CNN. Background pixels or equivalent are classified as such and ignored by the mechanical sorter. Each object is segmented and given a different pattern, which may correspond to a classification. Some objects overlap, and the CNN distinguishes them as different objects and gives them different patterns. The mechanical sorter may be instructed to treat these as different objects or a single group of objects and classify them accordingly.

[0068] The system further includes a mechanical sorter configured to sort the objects using the analysis of the processing circuit. The mechanical sorter may be at least one valve, a mechanical kicker, a robotic arm, or any other suitable device capable of mechanically moving objects.

[0069] The at least one valve is arranged to discharge at least one air stream that pushes the objects to a desired location. The desired location may be a different part of the bulk, a chute or container arranged in the direction of the at least one valve, or simply a bulk sorting system. The at least one valve is preferably a solenoid valve, as it has a relatively short switching time and a relatively high throughput performance.

[0070] The valves can be arranged in an array with distances between valves on the order of mm to cm. In one embodiment, the bulk sorting system includes at least 10 valves per meter. In other embodiments, more than 10 valves per meter (e.g., 12, 15, 20, 100, 250, etc.) are used. Depending on the size and weight of the object as identified by the CNN, multiple valves can be active at one time.

[0071] At least one mechanical kicker is positioned to kick objects to a desired location. The desired location may be a different portion of the bulk, a chute or container positioned in the direction of the at least one mechanical kicker, or simply the bulk sorting system. The kicker may be electromechanical, pneumatic, spring-driven, and / or hydraulic, although the kicker is preferably pneumatic. In one embodiment, the bulk sorting system includes at least 10 kickers per meter. In other embodiments, more than 10 kickers per meter (12, 15, 20, 100, 250, etc.) are used. The objects are sorted using the analysis of the processing circuitry, which may include directing a mechanical sorter to the objects identified by the CNN, which may further include sorting the objects according to the set to which they belong, the set being identified by the CNN.

[0072] The mechanical sorter can be configured to separate objects into at least two streams. This allows the bulk sorting system to sort objects into multiple streams in bulk, with each stream configured with a specific set of objects, for example. As an example, a bulk of trash can be sorted into a stream of plastics, a stream of metals, and a stream of other materials, with each stream directed or transported to a different location. The system can be used for bulk sorting of food and mining materials in a similar manner.

[0073] Mechanical sorters can be configured to eject unwanted objects from the bulk, allowing the bulk sorting system to remove unwanted objects from the bulk. The ejected objects are then sorted into specific containers or excluded from the remaining objects.

[0074] 4 shows an embodiment of a mechanical sorter that includes an array of valves 21 and one mechanical kicker 23. Several valves in the array are positioned on opposite sides of a chute 25. The chute is intended to separate a particular set of objects from the remaining bulk. When such objects are classified by the CNN, the processing circuitry can be configured to instruct several of the valves on the opposite side of the chute to activate when a classified object passes by the valve, a number proportional to the size or weight of the object identified by or using output from the CNN.

[0075] The mechanical sorter's mechanical kicker is positioned such that if an unwanted object is too large or heavy to be moved by the array of valves, it will be ejected by the mechanical kicker before moving beyond the mechanical sorter.

[0076] The bulk sorting system may further include using a conveyor belt 28 to transport the objects along the detection range of at least one optical sensor to a mechanical sorter. The objects can be transported in any number of ways, such as Brownian motion, fluid transport, or air current transport, but the inventors have found that a conveyor belt is the most efficient and does not interfere with optical measurement or sorting. The speed of the conveyor belt is approximately 2.5-3 m / s.

[0077] In other embodiments, the ejector is located on the same side as the chute, or free fall sorting, or a conveyor belt is used for material transport between detection and ejection.

[0078] The mechanical sorter can be positioned to sort the objects as they move along the conveyor belt. The sorter can be positioned near the conveyor belt and positioned to move the objects in a direction perpendicular to the direction of movement of the conveyor belt.

[0079] The mechanical sorter can be positioned to sort objects as they move along a chute or as they free fall. The chute or free fall can be positioned at the end of a conveyor belt or along the edge of a conveyor belt. In this embodiment, since the speed of the objects is relatively difficult to predict and control, it is a priority that the mechanical sorter be able to react quickly. In one embodiment, the mechanical sorter has an actuation accuracy of less than 1 millisecond for small particles and less than 10 milliseconds for larger particles such as bottles. Sorting can include, for example, moving objects to a desired location so that wanted objects are chute or fall into a different container than unwanted objects. The mechanical sorter can be arranged to sort the objects such that different overlapping and / or stacked objects are separated, as analyzed by the processing circuitry. Because bulk sorting typically does not involve pre-sorting, overlapping and stacking of objects is possible and very common. The processing circuitry may identify these objects, preferably using a CNN. Once identified, the mechanical sorter can be instructed or influenced by the processing circuitry to separate the objects. This can include, for example, opening an air valve at a precise time and precise moment.

[0080] The mechanical sorting may be arranged to sort the objects such that different overlapping and / or stacked objects are treated as a single group of objects as analyzed by the processing circuitry. Because bulk sorting does not typically involve pre-sorting or pre-processing to form a single stream of separated objects, overlapping and / or stacked objects are possible and quite common. The processing circuitry may preferably use a CNN to identify these objects.

[0081] Once identified, the mechanical sorter may not be able or willing to separate the objects, which may in turn depend on the mode of operation the processing circuitry is in. As such, different overlapping and / or stacked objects are treated as a single group that is processed as a single object. This processing may, for example, eject the group if it contains unwanted objects, or retain the group if it contains enough wanted objects. This may be measured using an appropriate measurement such as weight percentage or volume. Which action is taken and whether wanted or unwanted objects are prioritized depends on the configuration or mode of use of the processing circuitry / bulk sorting system. In other words, in one embodiment, overlapping and / or stacked objects are treated as a single group of objects, and for groups of objects that include at least a first object type and a second object type, the mechanical sorter is configured to sort the group of objects as either the first object type or the second object type based on the configuration of the bulk sorting system.

[0082] In one embodiment, the neural network does not handle overlapping objects differently than separated objects, but must learn to separate overlapping objects by providing negative feedback in the training process when overlapping objects are merged, only partially detected, or not detected at all. To handle this difficult task, many example images containing overlapping objects and corresponding labels (representing the contours of touching and overlapping objects) may be provided during the training of the CNN.

[0083] A process called data augmentation is typically used to increase the sample size by rotations, small scale changes, color changes, cropping, etc. applied to the entire image. Special augmentation techniques are used in conveyor belt setups to generate more distinct duplicates. By labeling all captured training data object instances with their class and surrounding contours, the images from the training process, as they were captured, can be used in the following new ways, using the standard augmentation techniques mentioned above:

[0084] Starting with an image captured from a conveyor belt without objects, labeled object instances are placed into this image and allowed to overlap. Objects that are completely occluded by other objects are removed from the labeled ground truth in the newly augmented image.

[0085] If an object is at least partially located behind another object, new image information for this area has to be generated according to the sensor properties - e.g. for NIR, VIS and RGB data the information of the upper object overwrites the existing information, for X-ray data the Beer-Lambert law is applied, for laser data a 3D height profile of the two objects is calculated and the scattering and intensity values ​​of the upper object are used.

[0086] The mechanical sorter can be set to work differently depending on the analysis of the processing circuitry. For example, a large, light object, as analyzed by the processing circuitry, will only use a portion of the possible ejectors since not all are needed for a light object, thereby saving energy. In an embodiment with an air valve array, only half of the air valves that cover the object will be used, thus saving energy and creating less turbulence. Reciprocally, the mechanical sorter may use more possible ejectors than normal to increase the sorter's yield and reduce sorting accuracy, which may be beneficial for heavy objects.

[0087] For mechanical sorters with variable strength, this can be further adjusted based on analysis of the processing circuitry, so that lighter objects may have a lower strength than heavier objects.

[0088] Additionally, the analysis of the processing circuitry can be used to find the center of gravity or boundaries of the object and influence the mechanical sorter to target these areas, which can be beneficial to increase the efficiency of the sorter and may be even necessary for certain types or sorters that require advanced targeting.

[0089] Furthermore, if the object's position is not stable, it may be beneficial to increase the number of ejectors activated and the time window for ejection to compensate for unpredictable motion.

[0090] FIG. 5a shows a bulk sorting system including a near-infrared (NIR) scanner 30 positioned to scan an object and an image sensor 32 positioned to capture image data of the object. The multispectral or hyperspectral data input to the CNN consists of the scan data and the image data. Although FIG. 5a and other figures show several types of optical sensors being used in tandem, each optical sensor can operate independently or together with any other optical sensor. The optical sensors can be further arranged to measure radiation of a radiation source or include a separate radiation source.

[0091] A NIR scan consists of spectroscopic data about the absorption of objects in this wavelength range. Other wavelengths can be used instead of or in addition to the NIR range, such as visible light, ultraviolet (UV) light, or X-rays. The NIR light can be provided by a radiant source or a separate light source. A scan can be represented as multispectral or hyperspectral data by acquiring a separate spectrum for each pixel or equivalent of the scanned area.

[0092] The image data may include, for example, RGB pixel values ​​of the reflected color of the object. This may be used for conventional image processing by processing circuits, such as object recognition and image segmentation. To allow for the measurement of the image data, the object may be illuminated by a radiation source. The image data may be represented as multispectral or hyperspectral data by obtaining separate values ​​for the amount of red, green, and blue for each pixel or equivalent of the scanned area.

[0093] In Fig. 5b, a bulk sorting system is shown that includes a laser triangulator 34 arranged to measure 3D data of the object and an electromagnetic detector 40 arranged to measure the electromagnetic properties of the object. The multispectral or hyperspectral data input to the CNN consists of the 3D data and the electromagnetic properties. It is noted that the electromagnetic detector is not an optical sensor, but can be combined with any optical sensor to obtain data from the object that is input to the CNN as multispectral or hyperspectral data.

[0094] 3D data can be measured using laser height intensity scanning. This consists of a laser pointed at an object and a collector, such as a camera, which measures the intensity of the laser. The collector is positioned to measure the intensity from a surface where the laser is pointed 10-20cm above the surface. This allows the collector to detect where the laser hits an object, because when an object perturbs the path of the laser, it affects its intensity.

[0095] 3D data can be measured using laser height intensity scanning. This consists of a laser that is pointed at an object, and a collector, such as a camera, that measures the intensity of the laser. The collector is positioned to measure the intensity from the surface, where the laser is pointed 10-20cm above the surface. This allows the collector to detect where the laser hits an object. This is because when an object perturbs the path of the laser, it affects its intensity.

[0096] The collector may be equipped with a bandpass filter to filter out ambient light so that only the intensity of the laser wavelength is measured.

[0097] The 3D data includes, for example, a mapping of the height of an object. The 3D data can be represented as multispectral or hyperspectral data by obtaining a height value for each pixel or equivalent of the scanned area.

[0098] Electromagnetic properties can be measured, for example, using the properties of reflection of an applied electromagnetic wave or the conduction of an applied current. This can be used, for example, to detect metallic objects. Arrays of single coils are used to detect metals, and arrays of balanced coils are used to distinguish between different metal types.

[0099] As in the case of optical measurements, it may not be possible to precisely and directly assign the measured electromagnetic properties to specific pixels. Therefore, the assignment may be estimated, possibly using the optical data to distinguish between objects. The electromagnetic properties may be represented as multispectral or hyperspectral data by assigning the measured electromagnetic properties to estimated pixels or equivalents of the scanned area.

[0100] In the embodiment of Figure 5b, the electromagnetic detector uses the multispectral or hyperspectral data from the laser triangulation device to assign measured electromagnetic properties to the multispectral or hyperspectral data with improved accuracy of the assignment. Communication of the multispectral or hyperspectral data to the electromagnetic detector is shown directly from the laser triangulation device, but may occur via processing circuitry. The assignment of the measured electromagnetic properties to the multispectral or hyperspectral data may instead be performed separately in the processing circuitry after some or all of the measured electromagnetic properties and the multispectral or hyperspectral data have been collected and before they are input into the CNN.

[0101] 6a shows a bulk sorting system including a laser scanner 36 with a rotating polygon mirror arranged to measure the laser scattering and / or anti-scattering properties of the object, a pulsed LED emitter 38 arranged to measure the light anti-scattering properties of the object, and an X-ray camera 42 arranged to measure the X-ray transmission of the object. The multispectral or hyperspectral data input to the CNN includes the laser scattering and / or anti-scattering properties, the light anti-scattering properties, and the X-ray transmission of the object.

[0102] Although the different optical sensors are shown as a single unit in FIG. 6a, they may be any number of units.

[0103] The laser scattering and / or anti-scattering properties of an object include information of the reflectance and absorption of the emitted laser light, which is used for example for object recognition. The laser scattering and / or anti-scattering properties of an object can be represented as multispectral or hyperspectral data by obtaining the laser scattering and / or anti-scattering properties of each pixel or equivalent of the scanned area.

[0104] The laser scattering and / or anti-scattering properties of an object are measured using a detector positioned to measure the intensity of laser light reflected by the object.

[0105] A laser scanner with a rotating polygon mirror emits a point laser at the rotating polygon mirror, which reflects the laser in different directions. This allows the measurement of multiple objects and their scattering properties in different directions without a target. The laser may have any wavelength, a combination of several wavelengths, or may alternate between different wavelengths. The laser can be considered as a radiation source or provided separately. The receiver may be configured to measure the reflected laser light, the scattered laser light, or both. Alternatively, the receiver may measure the fluorescence effect of the object caused by the laser.

[0106] The LED emitter may be set to emit several different wavelengths, such as six different colors. The LED emitter is preferably pulsed so that each color is emitted as a separate pulse. The LED emitter may be considered the radiation source or may be provided separately. LED emitters are beneficial in that they require little maintenance and calibration and generate relatively little heat.

[0107] The anti-light scattering properties of the object are measured by a detector, such as a line scan detector, preferably an InGaAs line scan detector, that measures the intensity of the reflected LED light. The anti-light scattering properties of each emission color and the background without the LED emission can be represented as multispectral or hyperspectral data by obtaining the anti-light scattering properties of each pixel or equivalent of the scanned area.

[0108] X-ray transmission involves the dielectric constant of an object to X-ray radiation and is measured by an X-ray camera placed opposite an X-ray emitter on the opposite side of the object. The dielectric constant of objects to X-ray radiation may be indicative of their atomic density and / or their thickness. The X-ray emitter may be considered as the radiation source or may be provided separately. X-ray cameras may have single, dual or multiple energy measurement ranges. X-ray transmission may be represented as multispectral or hyperspectral data by acquiring separate transmission spectra for each pixel or equivalent of the scanned area.

[0109] X-ray fluorescence allows the detection of elements present in particles. Materials are excited by low-energy X-ray radiation, which causes element-specific fluorescence to be emitted. Using an energy-dispersive X-ray sensor, this fluorescence can be measured and represented as multispectral or hyperspectral data. The result of the fluorescence is information about the presence of elements and their concentrations. 6b shows a bulk sorting system that includes a hyperspectral camera 44 positioned to scan an object, where the multispectral or hyperspectral data comprises the scan data. Hyperspectral cameras are beneficial in that they are an efficient way of collecting large amounts of data for use in CNNs.

[0110] Figure 7 shows a flow chart of the operation of the processing circuit according to one embodiment. Various data are collected from various sensors (note that the sensors included in Figure 7 are merely examples) and modified into multispectral or hyperspectral data. This multispectral or hyperspectral data is input to a CNN. The CNN outputs the detected and classified and / or semantically segmented multispectral or hyperspectral data for post-processing.

[0111] The data collected in the exemplary embodiment of FIG. 7 are HSI, which corresponds to the hyperspectral image captured by the hyperspectral camera 44; RGB, which corresponds to the red-green-blue image captured by the image sensor 32; Scatter, which corresponds to the laser scattering and / or anti-scattering characteristics captured by the laser scanner 36; Height, which corresponds to the 3D data captured by the laser triangulation device 34; and EM, which corresponds to the electromagnetic characteristics captured by the electromagnetic detector 40.

[0112] Post-processing involves converting the output into something more easily interpretable by a mechanical sorter, which may include setup instructions for the mechanical sorter that are created before sorting of the objects corresponding to the measurement data occurs.

[0113] Post-processing may include, for example, interpreting the classification to find out whether the detected objects are wanted or unwanted, and creating instructions for the mechanical sorter on how to process them. These instructions may include a control schedule for which mechanical ejectors to activate and when.

[0114] These post-processed instructions are then sent to a mechanical sorter which sorts the objects, which may include ejecting unwanted objects.

[0115] 8 shows a flow chart of a hybrid operation of a processing circuit including both a CNN and a conventional pattern recognition algorithm. In this embodiment, the processing circuit is configured to input at least a portion of the multispectral or hyperspectral data into a pattern recognition algorithm. The results of both the CNN and the pattern recognition algorithm are then used to sort objects by a mechanical sorter, which may or may not include post-processing of the results before feeding them to the mechanical sorter.

[0116] Hybrid operation is beneficial in that different information can be obtained by different analytical methods. Although only two methods are disclosed in Figures 8 and 9, any type and number of optical and / or electromagnetic analyses can be combined in this manner.

[0117] In FIG. 8, the spectral scan, RGB image, laser scatter and / or anti-scatter properties, and electromagnetic properties are input into a conventional pattern recognition algorithm individually without collating the different data into a single multispectral or hyperspectral data, although input of multispectral or hyperspectral data is also possible.

[0118] The data input to the different analysis methods can be the same or different, for example in Figure 8, the 3D data is input to a CNN and not to a traditional pattern recognition algorithm.

[0119] 9 shows a flow chart of a hybrid operation of the processing circuit using two computing units shown in dashed boxes. This embodiment uses different computing units for different types of data analysis, such that the computing unit on the left uses CNN and the computing unit on the right uses traditional pattern recognition and post-processing.

[0120] This is beneficial in that each computing unit can be optimized for a particular type of analysis, thereby increasing the efficiency of the analysis step and reducing the total time for sorting.

[0121] This separation can be implemented in a variety of ways, such as multiple computing units implementing the CNN analysis, or post-processing implemented in a separate computing unit. There is no limit to the number of computing units that can cooperate in the operation of the processing circuit, and hybrid operation is possible but not required.

[0122] A flow chart of a method for sorting objects into chunks is shown in Figure 10. The method 100 includes several steps that are performed in sequence.

[0123] The irradiating step 110 includes irradiating the object using at least one radiation source. The radiation source may be an LED, a light bulb, a laser, an X-ray emitter, and / or any other suitable radiation source. The radiation sources may be arranged as different units for ambient lighting and / or to radiate in different directions.

[0124] The capturing step 120 includes capturing reflected radiation of the object using at least one optical sensor, which may be a near-infrared (NIR) scanner, an image sensor, a laser triangulation sensor, a laser scanner, a pulsed LED emitter, and / or an X-ray camera.

[0125] The acquiring step 130 includes acquiring the reflected radiation as multispectral or hyperspectral data, which may include converting the captured radiation to multispectral or hyperspectral data. This step 130 may be performed in conjunction with the acquiring step 120 as the reflected radiation is captured.

[0126] Multispectral data may be multidimensional images where each pixel or equivalent (such as a superpixel or grouping of pixels) contains several bands (on the order of 1-10) with broad spectra (on the order of 100-1000 nm). Hyperspectral data may be multidimensional images where each pixel or equivalent (such as a superpixel or grouping of pixels) contains many bands (on the order of 4-10000) with narrow spectra (on the order of 1-50 nm).

[0127] The analyzing step 140 includes analyzing the reflected radiation of objects by inputting the multispectral or hyperspectral data into a Convolutional Neural Network (CNN) having at least two convolutional layers to detect and classify objects in the multispectral or hyperspectral data and / or to semantically segment the multispectral or hyperspectral data.

[0128] This step 140 is preferably performed by a processing circuit, which may include a processor, such as a central processing unit (CPU), a microcontroller, or a microprocessor, configured to execute program code stored in a memory to perform at least one step of a method for sorting objects into chunks. A sorting step 160 involves sorting the objects by a mechanical sorter using the results of the analyzing step 140. The mechanical sorter may be at least one valve, a mechanical kicker, a robotic arm, or any other suitable device capable of mechanically moving objects.

[0129] The method 100 may further include a post-processing step 150 which involves post-processing the results of the analysis step 140 to convert the results into ones more suitable for use in the sorting step.

[0130] Many modifications and other embodiments of the inventions described herein will come to mind to one skilled in the art to which these inventions pertain having the benefit of the teachings presented in the foregoing description and the associated drawings. It is therefore to be understood that the invention is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, while the foregoing description and the associated drawings describe example embodiments in the context of certain exemplary combinations of elements and / or features, different combinations of elements and / or features may fall within the scope of the appended claims. In this regard, for example, it is contemplated that combinations of elements and / or features different from those explicitly described above may also be set forth in some of the appended claims. Where an advantage, benefit, or solution to a problem is described herein, it is to be understood that such advantage, benefit, and / or solution may be applicable to some, but not necessarily all, exemplary embodiments. Thus, an advantage, benefit, or solution described herein should not be considered critical, necessary, or essential to all embodiments or those claimed herein. Although specific terms have been employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

1. A bulk sorting system for sorting a plurality of objects (1) in bulk, comprising: at least one radiation source (10) arranged to radiate onto said object; at least one optical sensor (12) arranged to capture reflected radiation (22) of the object and obtain the reflected radiation as multispectral or hyperspectral data (24), the optical sensor (12) including a near infrared (NIR) scanner (30) arranged to scan the object, the multispectral or hyperspectral data comprising scan data; a processing circuit (16) configured to analyze reflected radiation of the objects by inputting the multispectral or hyperspectral data into a convolutional neural network (CNN) (18) having at least two convolutional layers in order to detect and classify the objects in the multispectral or hyperspectral data; a mechanical sorter (20) that uses the analysis of the processing circuitry to sort the objects according to their classification, such that the different overlapping and / or stacked objects analyzed by the processing circuitry are treated as a single group of objects; Equipped with the at least one optical sensor comprises a laser scanner (36) with a rotating polygon mirror arranged to measure the laser scattering and / or anti-scattering properties of the object, a pulsed LED emitter (38) arranged to measure the light scattering properties of the object, and / or an X-ray camera (42) arranged to measure the X-ray transmission through the object; A bulk sorting system, wherein the multispectral or hyperspectral data includes laser scattering and / or anti-scattering properties, light anti-scattering properties, and / or X-ray transmission of the object, respectively.

2. the at least one optical sensor includes an image sensor (32) positioned to capture image data of the object, the multispectral or hyperspectral data including the image data; 2. The bulk sorting system of claim 1.

3. 3. The bulk sorting system of claim 1 or 2, wherein the at least one optical sensor comprises a multispectral or hyperspectral camera (44) positioned to scan the object, and the multispectral or hyperspectral data comprises the scan.

4. the at least one optical sensor includes a laser triangulation device (34) arranged to measure 3D data of an object; the multispectral or hyperspectral data includes the 3D data; The measurement of the 3D data may include a laser height intensity scan.

4. A bulk sorting system according to any one of claims 1 to 3.

5. and further comprising an electromagnetic detector (40) arranged to measure an electromagnetic characteristic of the object; The processing circuitry is further configured to analyze electromagnetic signatures by inputting the measured electromagnetic signatures into a CNN to detect and classify objects within the multispectral or hyperspectral data.

5. A bulk sorting system according to any one of claims 1 to 4.

6. 6. The bulk sorting system of claim 1, further comprising: transporting the objects along a detection range of the at least one optical sensor to the mechanical sorter using a conveyor belt.

7. 7. The bulk sorting system of claim 1, wherein the mechanical sorter is further configured to separate the objects into at least two streams and / or to eject unwanted objects from the bulk.

8. For a group of objects including at least a first object type and a second object type, the mechanical sorter is configured to sort the group of objects as the first object type or the second object type based on a configuration of the bulk sorting system.

8. A bulk sorting system according to any one of claims 1 to 7.

9. 9. The bulk sorting system of claim 1, wherein the mechanical sorter is positioned to target a center of gravity or a boundary of the object to be analyzed by the processing circuitry.

10. and further comprising post-processing the detected and classified multispectral or hyperspectral data to configure the mechanical sorter before sorting occurs. A bulk sorting system according to any one of claims 1 to 9,

11. the processing circuitry is further configured to input at least a portion of the multispectral or hyperspectral data into a pattern recognition algorithm; The results of both the CNN and the pattern recognition algorithm are used by the mechanical sorter to sort the objects according to their detection and classification verified by the pattern recognition algorithm.

11. A bulk sorting system according to any one of claims 1 to 10.

12. 1. A method for sorting a plurality of objects in groups, comprising: irradiating the object with at least one radiation source (110); capturing (120) reflected radiation of the object with at least one optical sensor, the capturing (120) of the scan with a multispectral or hyperspectral scanner (30); acquiring (130) the reflected radiation as the multispectral or hyperspectral data, including the scan; inputting the multispectral or hyperspectral data into a convolutional neural network (CNN) having at least two convolutional layers to detect and classify the objects in the multispectral or hyperspectral data; Analysing (140) the reflected radiation of the object; Sorting (160) the objects by a mechanical sorter according to the classification using the results of the analyzing step, such that different overlapping and / or stacked objects are treated as a single group of objects; Equipped with the at least one optical sensor comprises a laser scanner (36) with a rotating polygon mirror arranged to measure the laser scattering and / or anti-scattering properties of the object, a pulsed LED emitter (38) arranged to measure the light scattering properties of the object, and / or an X-ray camera (42) arranged to measure the X-ray transmission through the object; The method, wherein the multispectral or hyperspectral data comprises laser scattering and / or anti-scattering properties, light anti-scattering properties, and / or X-ray transmission, respectively, of the object.

13. the step of capturing (120) reflected radiation of the object with at least one optical sensor includes capturing (120) a scan with an NIR scanner (30); 13. A method for sorting a plurality of objects in groups according to claim 12.

Images