Method and system for training a neural network to classify objects in a bulk flow

Abstract

The present disclosure relates to a method (100) for training a neural network (NN) stored on a computer-readable storage medium (10) to classify objects (AE) in a bulk flow, the method comprising the steps of: providing (101) input image data (50*) depicting objects (A*-E*) to be classified, the input image data (50*) being acquired by input imaging sensors (30, 30a-30e) of a first sensor technology design; and providing (102) auxiliary image data (50**), the auxiliary image data (50**) being acquired by auxiliary imaging sensors (40, 40a-40e) of a second sensor technology design; and providing (103) auxiliary image data (50**) according to a predetermined classification scheme. The method (100) includes a step (102) of providing representations of identical or similar objects (A**-E**) to be classified accordingly, and a step (103) of training the neural network (NN) stored in the computer-readable storage medium (10) by a processing unit (20) to classify the represented objects (A*-E*) in the input image data (50*) based on the classification of the represented objects (A**-E**) in the auxiliary image data (50**), wherein the represented objects (A*-E*) in the input image data (50*) correspond to the objects (AE) in the bulk flow, and the second sensor technology design is different from the first sensor technology design.

Description

Technical Field

[0001] The present invention relates to the field of classifying objects in a bulk flow. In particular, the present invention relates to a method and system for training a neural network stored in a computer-readable storage medium to classify objects in a bulk flow, and also to a system for implementing such a trained neural network in a neural network implementation sensor system for classifying and selectively sorting objects in a bulk flow.

Background Art

[0002] Automatic classification is a research field of topics that is carried out, for example, for recycling, mining, or food processing. In the implementation of recycling, classification techniques are used when sorting a mixture of garbage into the correct recycling containers. With the progress of technology, this classification and the resulting sorting can be performed more accurately and quickly than before.

[0003] U.S. Patent Application Publication No. 2018 / 243800 discloses a machine learning system for sorting a stream of single objects, which enables accurate classification of the objects when being sorted. However, such a technique is slow because it can only process a stream of single objects at a time.

[0004] Furthermore, in order to achieve sufficient classification results using a machine learning system, it is important that the sensor data used by the machine learning system is of high quality. This consequently leads to incorporating expensive sensors, which naturally hinders the machine learning system from being economically realistic for classification applications, especially sorting applications.

[0005] Therefore, improvement in this regard is necessary.

Disclosure of the Invention

[0006] One of the objectives of the present invention is to provide an improved solution that alleviates the disadvantages of the mentioned current solutions. This objective is achieved according to a new technological invention for training a neural network stored in a computer-readable storage medium. The new neural network implementation sensor system implements such a trained neural network and a method for sorting objects in a bulk flow by the trained neural network. All of these are defined in the appended independent claims, and the preferred embodiments are defined in the related dependent claims.

[0007] In a first aspect of the invention, this is provided by a method for training a neural network stored in a computer-readable storage medium to classify input image data with respect to the depicted object. The method includes providing input image data depicting the object to be classified, where the input image data is acquired by an input imaging sensor of a first sensor technology design; providing auxiliary image data, where the auxiliary image data is acquired by an auxiliary acquisition sensor of a second sensor technology design and depicts the same or similar objects that are classified according to a predetermined classification method; and training the neural network stored in the computer-readable storage medium by a processing unit to classify the object depicted in the input image data based on the classification of the object depicted in the auxiliary image data. The object depicted in the input image data corresponds to an object in a bulk flow, and the second sensor technology design is different from the first sensor technology design.

[0008] The advantage of this learning method is that it can provide a learned neural network that can classify the objects depicted in the input image data more efficiently and reliably than conventional solutions. This is achieved by using the differences in the image data provided by imaging sensors of different first and second sensor technology designs. Thereby, depictions in the input image data that are difficult or impossible to classify can be classified when associated with the classified depictions in the auxiliary image data.

[0009] As an example for explaining this operating principle in detail, assume that an input imaging sensor of a first sensor technology design is configured to provide input image data of one real object A that is depicted as a depiction A* of the object at a certain resolution, and an auxiliary imaging sensor of a second sensor technology design is configured to provide auxiliary image data depicting the same real object A as a depiction A** of the object at a resolution higher than the resolution of the depiction A* of the object. The higher resolution enables the correct classification of the depiction A** of the object as the real object A. Next, assuming that the identified depiction A* of the object in the input image data can be identified as a depiction A* of a certain object based on certain characteristic properties that are contrasted with the environment in which it is depicted, even if the depiction A* of the object in the input image data does not allow for the direct correct classification as the real object A, it is associated with the classified depiction A** of the object, enabling the depiction A* of the object to be classified more accurately by reference.

[0010] Thus, when properly learned by this learning method, the neural network is configured to identify the depicted object A* as object A without requiring assistance from the classification of the object depicted in the auxiliary image data. The neural network may be learned to associate the depiction of the object in the input image data, where it is difficult to correctly classify the depiction of the object due to image quality, with the depiction in the auxiliary image data, thus enabling the classification of the depicted object in the input imaging data. Further, the classified auxiliary image data can cause the neural network to learn to clarify the object depicted in the input image data.

[0011] The above-described embodiments merely show different image resolutions, but this principle of operation applies when the input imaging sensor and the auxiliary imaging sensor are selected from different sensor technology designs, for example, when the input imaging sensor and the auxiliary imaging sensor are of different general sensor technology designs, such as RGB sensor technology and near-infrared sensors, or of the same general sensor technology design, such as RGB sensor technology, but with qualitative differences available for the same general sensor technology, such as different resolutions. Thus, the input imaging sensor of the first sensor technology design and the auxiliary imaging sensor of the second sensor technology design may be sensors of different general sensor technologies or sensors of the same general sensor technology design but with different qualitative differences.

[0012] Regarding a neural network stored in a computer-readable storage medium, it may be an artificial neural network having an input layer and an output layer. Further, it may include one or more hidden layers between the input layer and the output layer. Each neuron in the input layer may correspond to a predetermined region of an image. Furthermore, a computer-readable storage medium may mean a medium that can store data in a format readable by a computer, that is, a mechanical device having computing power. The computer-readable storage medium may be, for example, disk storage, a memory card, a USB, an optical disk, volatile or non-volatile memory, etc. The neural network stored in the computer-readable storage medium may be learned in real time when image data is acquired from a bulk flow. Alternatively, the neural network stored in the computer-readable storage medium may be learned separately from the moment when image data is acquired from the bulk flow in a virtual environment provided by a computing device such as a computer, a mobile device, or a server.

[0013] Furthermore, the object to be depicted may be an object in a bulk flow. A bulk flow is an object flow, which may mean that the object is a bulk. Also, the flow of the object may include a case where the object is instantaneously and spatially separated from other objects so that the object is depicted as a single object of the bulk flow in the field of view of the imaging sensor. The imaging sensor can acquire image data of an object in the bulk flow when the object is instantaneously stopped or moving. Moreover, when depicted by an input imaging sensor or an auxiliary imaging sensor, the objects in the bulk flow may overlap each other as seen in the image data and may consequently block each other to some extent. The neural network can be learned to classify the objects in the bulk flow well even when they overlap each other, for example, by incorporating information from multiple sensors of the same or different sensor technology designs.

[0014] Furthermore, it is recognized that this learning method can be applied to train a neural network to classify generally spatially separated objects that do not move in bulk but move as individual objects in the corresponding object flow, and it should be understood that the invention in all aspects and embodiments disclosed herein can be applied with the necessary modifications for such uses.

[0015] Also, it is worth mentioning that the term "image data" can mean data that can be represented as an image, such as an image or distance information from a sensor to various points of an object. Therefore, the term "depicted object" means an object visible in the image data.

[0016] Also, as disclosed, the depicted objects in the auxiliary image data acquired by the auxiliary imaging sensor are classified according to a predetermined classification method. The predetermined classification method may be a classification method including a step of identifying characteristic features in the auxiliary image data and a step of classifying a set of the identified characteristic features as an object with good accuracy. Here, good accuracy may mean at least, for example, 95%, or 99% or more, or 100% accuracy. For example, 99% accuracy may mean that for every 100 classification trials, 1 of these classifications was statistically incorrect in some way. 100% accuracy means that for every 100 classification trials, 0 of these were statistically incorrect in any way, or that for every 1000, 10000, 100000, etc. classification trials, only 1 statistically incorrect classification occurred.

[0017] Furthermore, the expression "classifying the depicted object" may mean that when the image data depicts object A, it is determined that the classified image data depicts object A. Alternatively, this may mean that the image data is classified as image data depicting object A.

[0018] Furthermore, when the image data depicts objects A and B, the expression "classify the depicted objects" may mean that it is determined that the image data depicts the classified objects A and B, or that the image data depicts a combination of the classified objects A and B. Alternatively, this may mean that the image data is classified as image data depicting object A or B, or that the image data is classified as image data depicting a combination of objects A and B.

[0019] Furthermore, the expression "classified auxiliary image data depicting the same or similar objects" may mean that the image imaging sensor and the auxiliary imaging sensor depict the same one or more objects simultaneously or at various instants in each detection area. Also, this can mean auxiliary image data depicting similar classified objects, and thus the classified objects or the classified auxiliary image data are accessible to a library of classified auxiliary image data.

[0020] According to one embodiment, the second sensor technology design is a sensor technology design capable of providing higher quality image data and / or auxiliary image data not provided by the first sensor technology design than the first sensor technology design. Thereby, the neural network can be learned to be more efficient and reliable in classifying objects in bulk flow. When the second sensor technology design is appropriately selected, objects that are difficult to classify by nature due to texture or other characteristics can be more easily classified by the neural network when learned by the classification of the objects depicted in the auxiliary image data resulting from the selected second sensor technology design. For example, the neural network can be learned to be able to classify the objects depicted in the input image data acquired by the RGB sensor based on the classification of the objects depicted in the auxiliary image data acquired by the near-infrared (NIR) sensor or the X-ray sensor. Therefore, the learned neural network can be configured to classify the depicted objects well using a much cheaper sensor.

[0021] According to one embodiment, the step of training the neural network stored in the computer-readable storage medium is further based on additional non-image data of the object to be classified or data specified by the user. Non-image data can mean weight measurement, volume measurement, porcelain detection, or measurement of sound (e.g., from the bounce or movement of an object). This can further improve the neural network to better classify objects in bulk flow.

[0022] According to one embodiment, this method includes providing a neural network stored in a computer-readable storage medium in a neural network-implemented sensor system configured to acquire input image data of bulk flow by at least one input imaging sensor of the first sensor technology design. The neural network may be trained when provided to the neural network-implemented sensor system.

[0023] According to one embodiment, a neural network implemented sensor system is provided to monitor the operation of a classification and / or sorting system configured to obtain auxiliary image data of a bulk flow by at least one auxiliary imaging sensor, the classification and / or sorting system comprising means for classifying objects depicted in the auxiliary image data according to a predetermined classification scheme. Thereby, the neural network implemented in the neural network implemented sensor system can be learned in real time based on the information obtained by the step of monitoring the classification and / or sorting system. The classification and / or sorting system may be configured to improve the quality of the classification provided by the classification means, and the neural network of the neural network implemented sensor system may be learned accordingly, but using the input imaging sensor instead of the auxiliary imaging sensor. Therefore, higher cost efficiency can be provided.

[0024] According to one embodiment, the method comprises providing first and at least second input image data depicting an object to be classified, the first and at least second input image data being obtained by respective first and at least second input imaging sensors of a first sensor technology design, the first input imaging sensor being configured to obtain input image data of an object in a bulk flow in a first detection area, and at least the second input imaging sensor being configured to obtain input image data of an object in a bulk flow in at least a second detection area, the at least second detection area being different from the first detection area, and the step of training a neural network stored in a computer-readable storage medium further comprises a step based further on a difference with the first and at least second input image data.

[0025] The detection area may mean a general area where the imaging sensor acquires image data. This may mean an area surrounded by the field of view of the imaging sensor. Moreover, the neural network may be trained based on image data corresponding to two or more detection areas, for example, 3, 4, 5, 6, 7, 8, 9, or 10 detection areas. One or more detection areas may be designated to detect specific objects in the bulk flow. The bulk flow may be sorted in relation to each detection area. Two consecutive detection areas may be arranged at a predetermined distance from each other. The sorting between two detection areas may be automatically performed by the robot, but in some embodiments, there may be human intervention to sort specific objects. Thus, the system can also be partially learned by a person who modifies the bulk flow between two detection areas, for example, removing a specific object from the bulk flow.

[0026] The expression "learning based on the difference between the first and at least the second input image data..." may mean that the neural network can infer the classification of the objects depicted in the other input imaging data by considering the objects normally classified from one of the input imaging data, thereby improving the classification in more complex situations. As an example, assume that when depicted in the first acquired input image data, the object in the first detection area is somewhat unclear due to other objects. Subsequently, when it moves to the next detection area, the object moves, thereby enabling an accurate classification of the object. In combination with this and information regarding objects removed or added from the bulk flow between the first detection area and the next detection area, the classification of the unclear object can be inferred, and thus the neural network can be trained to classify somewhat unclear objects.

[0027] Furthermore, information on how the bulk flow changes between the first and second detection areas can be provided, which can facilitate the learning of the neural network.

[0028] Furthermore, one or more of the first and at least the second input imaging sensors may be of a different sensor technology design. This may enable more general classification along a single stream of bulk flow.

[0029] According to one embodiment, at least two of the first and at least the second input imaging sensors are of different sensor technology designs. Thereby, the neural network can be trained to easily classify objects. For example, at least two input imaging sensors can be provided to acquire input image data in two consecutive detection areas. Thus, the neural network can be trained considering information about the depicted objects, which are depicted differently in each input image data. This can further reduce the ambiguity in the classification of a particular object and improve the neural network being trained.

[0030] According to one embodiment, the first sensor technology design and the second sensor technology design are selected from the group of sensor technology designs including near-infrared sensors, X-ray sensors, CMYK sensors, RGB sensors, volume measurement sensors, spectroscopic point measurement systems, visible light spectroscopy, near-infrared spectroscopy, mid-infrared spectroscopy, X-ray fluorescence sensors, electromagnetic sensors, laser sensors such as laser triangulation systems or scanning lasers for scattered lasers, multi-spectral systems using LEDs / pulsed LEDs / lasers, laser-induced breakdown spectroscopy (LIBS), fluorescence detection, detectors for visible or invisible markers, transmission spectroscopy, transflectance / intreractance spectroscopy, softness measurement, and thermal cameras, and / or the first sensor technology design and the second sensor technology design are of the same general sensor technology design but have different qualitative differences.

[0031] According to a second aspect of the present invention, a neural network implemented sensor system is provided. The neural network implemented sensor system comprises one or more input imaging sensors configured to acquire input image data of an object in a bulk flow, and a computer-readable storage medium storing a learned neural network learned by the method according to the first aspect of the present invention or any embodiment thereof.

[0032] The neural network may be learned with particular reference to such a neural network implemented sensor system. This may enable more efficient classification processing. Further, the neural network implemented sensor system may be modular with respect to the complexity and size of the bulk flow. Thus, the neural network may be configured for a desired application.

[0033] The neural network implemented sensor system may be communicatively coupled to a processing device, such as a computer, mobile device, server, etc., comprising a processor configured to process the classification of the depicted object in the input image data. The neural network implemented sensor system may comprise a processing unit. The neural network implemented sensor system may comprise a plurality of input imaging sensors of a first sensor technology design. The input imaging sensors may be of different sensor technology designs. The input imaging sensors may be configured to acquire input image data of an object in a bulk flow in different detection areas. Two or more input imaging sensors may be configured to provide input image data, for example, in different directions, or combined into one input image data corresponding to the detection area, to acquire input image data in the same detection area.

[0034] In one embodiment, the neural network-implemented sensor system is configured to be disposed in a bulk flow distribution system and thereby classify objects in the bulk flow being distributed. The "distribution system" may mean, for example, a system for distributing a bulk flow by a conveyor belt. Such a distribution system may be configured to be branched into smaller bulk flows and / or a plurality of branches of the bulk flow may be combined into larger bulk flows. The neural network-implemented sensor system may be configured to classify objects in the bulk flow at each detection area along the distribution system. The classification determined by the neural network-implemented sensor system may be transferred to a monitoring terminal of the distribution system, whereby the classified objects are presented to enable monitoring of the objects in the bulk flow.

[0035] In one embodiment, the neural network-implemented sensor system is configured to be disposed in a bulk flow sorting system and thereby sort objects in the bulk flow by one or more sorting units based on classifications provided by a learned neural network. The "bulk flow sorting system" or "sorting system" may mean a distribution system having sorting capabilities.

[0036] In one embodiment, a neural network implemented sensor system is configured to share a neural network stored in a computer-readable storage medium with a second neural network implemented sensor system. Thereby, a single neural network can be learned and subsequently distributed to at least one other neural network implemented sensor system. Thus, it is not necessary for each individual neural network implemented sensor system to be learned, and a single learned neural network can be shared among multiple neural network implemented sensor systems. This can ensure that multiple neural network implemented sensor systems all operate similarly in the classification operation aspect. "Sharing" can mean that the neural network is copied and stored in the computer-readable storage medium of each of the multiple neural network implemented sensor systems. Alternatively, "sharing" can mean that the neural network implemented sensor system accesses a neural network stored in a common computer-readable storage medium.

[0037] In one embodiment, one or more input imaging sensors are selected from the group of sensor technology designs including near-infrared sensors, X-ray sensors, CMYK sensors, RGB sensors, volume measurement sensors, spectroscopic point measurement systems, visible light spectroscopy, near-infrared spectroscopy, mid-infrared spectroscopy, X-ray fluorescence sensors, electromagnetic sensors, laser sensors such as laser triangulation systems or scanning lasers for scattered lasers, multi-spectral systems using LEDs / pulsed LEDs / lasers, laser-induced breakdown spectroscopy (LIBS), fluorescence detection, detectors for visible or invisible markers, transmission spectroscopy, transflectance / intrareflectance spectroscopy, softness measurement, and thermal cameras, and / or at least two of the multiple input imaging sensors are of the same general sensor technology design but have different qualitative differences.

[0038] According to a third aspect of the present invention, there is provided a method of sorting objects in a bulk flow by a sorting system. The sorting system comprises a neural network implemented sensor system according to the second aspect of the present invention or any embodiment thereof. The method includes obtaining input image data of an object in a bulk flow in a first detection area by at least one input imaging sensor of a first sensor technology design, classifying the object depicted in the input image data using a classification output by a learned neural network of a computer-readable storage medium by a processing unit, and sorting the object in the bulk flow by a sorting unit based on how the depicted object is classified. Thereby, the object can be sorted efficiently, measurably, and reliably. The method can be applied to sort objects by a plurality of sorting units each arranged in or following a detection area. Each detection area may be configured to detect a particular type of object in the bulk flow, and as a result, following each detection area, or following a series of detection areas operating in parallel to detect a particular object type, the type of object in the bulk flow is reduced.

[0039] The present invention is defined by the appended independent claims, and embodiments are defined by the appended dependent claims, the following description, and the drawings.

Brief Description of the Drawings

[0040] The present invention will be described in more detail below with reference to the accompanying drawings.

Figure 1

Figure 2

Figure 3a

Figure 3b

Figure 4a

Figure 4b

Figure 4c

Figure 5a

Figure 5b

Figure 6

Figure 7

Figure 8

Figure 9

Description of Reference Numerals

[0041] NN - Neural Network TNN - Trained Neural Network A - E - Objects in bulk flow A* - E* - Objects depicted in the input image data Objects depicted in the A*-E* - auxiliary image data 10 - Computer-readable storage medium 20 - Processing unit 30, 30a - 30e - Input imaging sensor 40, 40a - 40e - Auxiliary imaging sensor 50, 50a - 50e - Detection area 50* - Input image data 50** - Auxiliary image data 60 - Neural network implementation sensor system 70 - Classification and / or sorting system 71 - Means for classifying objects 72 - Sorting unit 73 - Means for transportation 74 - Terminal 100 - Learning method 101 - 105 - Steps of the learning method 200 - Sorting method 201 - 203 - Steps of the sorting method Detailed description of the invention

[0042] The present invention will be described below with reference to the accompanying drawings showing preferred embodiments of the present invention. However, the present invention can be embodied in many different forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope of the invention to those skilled in the art. The terminology used in the detailed description of the specific embodiments shown in the accompanying drawings is not intended to limit the invention. In the drawings, like numbers refer to like elements.

[0043] Figure 1 shows a flowchart of learning method 100. Thereby, the neural network NN stored in the computer-readable storage medium 10 is trained by the learning method 100 executed by the processing unit 20 to classify objects A - E in the bulk flow. As a result, as seen in Figure 2, the neural network NN stored in the computer-readable storage medium 10 becomes the trained neural network TNN. This trained neural network TNN can then be easily used in various scenarios related to classifying objects A - E in the bulk flow.

[0044] Figure 3a shows a setup of how the learning method 100 is performed in one embodiment. The neural network NN stored in the computer-readable storage medium 10 is trained by the learning method 100 executed by the processing unit 20. Both input image data 50* and auxiliary image data 50** are used in the learning method 100. The input image data 50* is acquired by the input imaging sensor 30. The auxiliary image data 50** is acquired by the auxiliary imaging sensor 40. Both the input imaging sensor 30 and the auxiliary imaging sensor 40 are configured to acquire the image data of objects A and B in the bulk flow passing through the detection area 50 where the image data is acquired, as seen in Figure 3a. It is not necessary for the input image data 50* and the auxiliary image data 50** to be acquired simultaneously. Rather, in some embodiments, as seen in Figure 3b, the input imaging sensor 30 may be configured to acquire the input image data 50* of objects A and B in the bulk flow at the first detection area 50a, and the auxiliary imaging sensor 40 may be configured to acquire the auxiliary image data 50** of objects A and B at the second detection area 50b.

[0045] Moreover, the learning method does not need to be performed when the input image data 50* and the auxiliary image data 50** are acquired. The learning method 100 may be performed later based on the image data 50* and 50** stored in a computer-readable storage medium. Moreover, the auxiliary image data 50* may be image data already acquired from similar objects.

[0046] Returning to FIG. 1, the learning method includes step 101 of providing input image data 50* depicting the objects A*-E* to be classified, and the input image data 50* is acquired by the input imaging sensor 30 of the first sensor technology design. An example of the input image data is shown in FIG. 4b depicting the actual objects A-E of the bulk flow in the detection area 50. As shown in FIG. 4b in different patterns, the depiction of the objects A*-E* may not be an appropriate representation of the actual objects A-E, which may be the result of the limitations and / or characteristics of the first sensor technology design.

[0047] Furthermore, the learning method includes step 102 of providing auxiliary image data 50**, and the auxiliary image data 50** is acquired by the auxiliary imaging sensor 40 of the second sensor technology design, and the auxiliary image data 50** depicts the same or similar objects A**-E** classified according to a predetermined classification method. An example of the auxiliary image data depicting the actual objects A-E of the bulk flow in the detection area 50 is shown in FIG. 4c. As shown in FIG. 4c, the depicted objects A**-E** have a pattern more similar to the pattern of the actual objects A-E in the detection area 50, which indicates that the image quality of the auxiliary image data enables more accurate classification according to a predetermined classification method.

[0048] Furthermore, the learning method 100 includes a step of causing the processing unit 20 to train a neural network NN stored in the computer-readable storage medium 10 to classify objects A*-E* depicted in the input image data 50* based on the classification of objects A**-E** depicted in the auxiliary image data 50**. In one embodiment, this is made possible by associating the depiction of objects A*-E* in the input image data 50* with the depiction of objects A**-E** in the auxiliary image data 50**, which are classified in order. Thus, the trained neural network TNN can be configured to accurately classify the depiction of objects A*-E* even when the image quality of the input image data 50* is such that it is difficult to accurately classify the depicted objects A*-E*.

[0049] Obviously, the learning method 100 is for training the neural network NN to classify objects A-E in a bulk flow. However, the learning method 100 can also be applied to training the neural network NN to classify objects A-E in a production line where the objects are arranged in order, for the purpose of, for example, detecting defective products based on predetermined conditions.

[0050] Also, the second sensor technology design is different from the first sensor technology design. In some embodiments, the second sensor technology design is a sensor technology design capable of providing higher-quality image data and / or auxiliary image data not provided by the first sensor technology design than the first sensor technology design.

[0051] In some embodiments, the learning method 100 includes step 104 of providing a neural network NN stored in a computer-readable storage medium 10 in a neural network implementation sensor system 60 configured to obtain bulk input image data 50* by at least one input imaging sensor 30 of a first sensor technology design. This is shown in FIG. 5a. The neural network implementation sensor system 60 includes a computer-readable storage medium 10 that stores the neural network NN to be learned, a processing unit 20 configured to execute the learning method, and one input imaging sensor 30 configured to obtain input image data 50* of objects A and B in a detection area 50. FIG. 5a shows a neural network implementation sensor system 60 having only one input imaging sensor 30, but the neural network implementation sensor system 60 may include a plurality of input imaging sensors. The plurality of input imaging sensors may be provided to obtain input image data in different detection areas. In some embodiments, two or more of the plurality of input imaging sensors may be provided to obtain input imaging data of an object in bulk flow in the same detection area. The plurality of input imaging sensors may be of the same sensor technology design, or any combination thereof.

[0052] In addition, in FIG. 5a, the neural network implemented sensor system 60 is provided to monitor the operation of a classification and / or sorting system 70 configured to obtain auxiliary image data 50 of a bulk flow by at least one auxiliary imaging sensor 40. The classification and / or sorting system 70 further comprises means 73 enabling a bulk flow of objects, which in some embodiments is a conveyor belt system. Further, the classification and / or sorting system 70 comprises means 71 for classifying objects A - E depicted in the auxiliary image data 50 according to a predetermined classification scheme. The means 71 may be a processing unit or may comprise a processing unit configured to execute such a predetermined classification scheme. As shown in FIG. 5a, the processing unit 20 of the neural network implemented sensor system is configured to access or receive the corresponding classification of the auxiliary image data 50 or the depiction A - E of its objects, enabling the neural network stored in the computer-readable storage medium 10 to be trained as a result.

[0053] FIG. 5b shows a neural network implemented sensor system 60 provided with a trained neural network TNN. The neural network implemented sensor system 60 is configured by the trained neural network TNN to well classify objects A and B without the need for assistance from the auxiliary imaging sensor 40. The neural network implemented sensor system 60 enables a classification method 100 for classifying objects A and B in the bulk flow. This classification method 100 includes the steps of obtaining input image data 50 of objects A and B in the bulk flow in a first detection area 50, and using the input image data 50 as an input to the trained neural network TNN to classify objects A and B by the trained neural network TNN.

[0054] FIG. 6 shows a plurality of neural network-implemented sensor systems 60a, 60b according to an embodiment. The first neural network-implemented sensor system 60a is configured to share a learned neural network TNN stored in a computer-readable storage medium 10 with a second neural network-implemented sensor system 60. Subsequently, the second neural network-implemented sensor system 60b is configured to classify objects A and B according to a classification method 100* by the learned neural network TNN. The plurality of neural network-implemented sensor systems 60a, 60b may comprise two or more, for example, two to ten or more such systems, and each system is provided to classify and / or sort objects in a bulk flow.

[0055] FIG. 7 shows an embodiment in which a neural network-implemented sensor system 60 is arranged in a sorting system 70 comprising a sorting unit 72 configured to sort objects into different streams OUT1, OUT2, the sorting being based on the classification of the objects. The neural network NN of the neural network-implemented sensor system 60 may be learned by a learned neural network TNN by the disclosed learning method 100.

[0056] FIG. 8 shows the layout of the neural network implemented sensor system 60 and the sorting system 70 with increased complexity. The sorting system 70 includes a plurality of sorting units 72 configured to sort objects based on the classification of the objects so as to move towards different output destinations OUT1, OUT2, OUTn-1, OUTn. The neural network implemented sensor system includes a fusion sensor 30 with two different sensor technology designs. In a preferred embodiment, this fusion sensor 30 includes an RGB sensor 31 and a volume measurement sensor 32. The neural network implemented sensor system 60 includes additional input imaging sensors 33-38 configured to acquire input image data in various streams from the sorting units 72. In one embodiment, one or more streams of the bulk flow are enabled by the conveyor belt 73.

[0057] Furthermore, in one embodiment, the other input imaging sensors 33-38 may also be RGB sensors.

[0058] However, depending on the specification of which objects are to be classified and / or sorted, other sensor technology designs may be desirable. One or more of the input imaging sensors 30, 30a-30e may be selected from the group of sensor technology designs including near-infrared sensors, X-ray sensors, CMYK sensors, RGB sensors, volume measurement sensors, spectroscopic point measurement systems, and / or the plurality of input imaging sensors may have the same general sensor technology design but different qualitative differences such as resolution.

[0059] Furthermore, although not shown, the sorting system 70 may be provided with an auxiliary imaging sensor 40 for acquiring auxiliary image data 50** for use in classifying objects A, B according to a predetermined classification method.

[0060] FIG. 9 shows a flowchart of a sorting method for sorting objects A-E in a bulk flow by a sorting system 70 including the neural network implementation sensor system 60 as described above. This method includes step 201 of obtaining input image data 50* of objects A-E in a bulk flow in first detection areas 50a-50e by at least one input imaging sensor 30a-30e of a first sensor technology design. Sorting method 200 further includes step 202 of classifying objects A*-E* depicted in the input image data 50* using a classification output by a learned neural network TNN of a computer-readable storage medium 10 by a processing unit. The sorting method includes step 203 of sorting objects A-E in a bulk flow by a sorting unit 72 based on how the depicted objects A*-E* are classified.

[0061] In the drawings and the specification, preferred embodiments and examples of the present invention are disclosed, and specific terms are employed, but these are used only in a general and illustrative sense and not for the purpose of limitation. The embodiments described with reference to the drawings are specific preferred embodiments and are described in consideration of specific aspects, and further embodiments may be provided by combining these embodiments. However, the scope of the present invention is defined by the following claims.

Claims

1. In a bulk flow, i.e., the flow of an object when the object is in bulk, a method (100) for training a neural network (NN) stored in a computer-readable storage medium (10) to classify objects (A-E), comprising: providing (101) input image data (50*) depicting the objects (A*-E*) to be classified, wherein the input image data (50*) is acquired by an input imaging sensor (30, 30a-30e) of a first sensor technology design; providing (102) auxiliary image data (50**), wherein the auxiliary image data (50**) is acquired by an auxiliary imaging sensor (40, 40a-40e) of a second sensor technology design and depicts objects (A**-E**) that are classified according to a predetermined classification scheme; training (103) the neural network (NN) stored in the computer-readable storage medium (10) by a processing unit (20) to classify the objects (A*-E*) depicted in the input image data (50*) based on the classification of the objects (A**-E**) depicted in the auxiliary image data (50**); wherein the objects (A*-E*) depicted in the input image data (50*) correspond to the objects (A-E) in the bulk flow; the second sensor technology design is different from the first sensor technology design; the input imaging sensor (30, 30a-30e) and the auxiliary imaging sensor (40, 40a-40e) depict the objects (A*-E*) at various instants of each detection area (50a, 50b); method (100).

2. The method (100) according to claim 1, wherein the second sensor technology design is a sensor technology design capable of providing higher quality image data and / or auxiliary image data not provided by the first sensor technology design than the first sensor technology design.

3. The step (103) of training the neural network (NN) stored in the computer-readable storage medium (10) is the method (100) according to claim 1, further based on additional non-image data of the object (A-E) to be classified or data specified by the user.

4. The method according to claim 1, including the step (104) of providing the neural network (NN) stored in the computer-readable storage medium (10) in a neural network implementation sensor system (60) configured to obtain the input image data (50*) of the bulk flow by at least one of the input imaging sensors (30, 30a-30e) among the first sensor technology designs.

5. The method according to claim 4, wherein the auxiliary image data (50**) of the bulk flow is obtained by at least one auxiliary imaging sensor (40, 40a-40e) of the classification and / or sorting system (70), and the classification and / or sorting system (70) comprises means (71) for classifying the object (A**-E**) depicted in the auxiliary image data (50**) according to the predetermined classification method.

6. The step (104) of providing first and at least second input image data (50*) depicting the object (A*-E*) to be classified, wherein the first and at least second input image data (50*) are obtained by each first and at least second input imaging sensor (30a-30e) of the first sensor technology design, the first input imaging sensor is configured to obtain the input image data (50*) of the object (A-E) in the bulk flow in the first detection area (50a), and the at least second input imaging sensor (30b-30e) is configured to obtain the input image data (50*) of the object (A-E) in the bulk flow in at least the second detection area (50b). The at least second detection region (50b - 50e) is different from the first detection region (50a), and the step (103) of training the neural network (NN) stored in the computer-readable storage medium (10) uses the normal classification of the object (A* - E*) depicted from the first input image data among the first and at least second input image data (50*), and includes inferring the depiction in the second input image data among the first and at least second input image data (50*), step (104) The method (100) according to claim 1, including this

7. The method (100) according to claim 6, wherein at least two of the first and at least second input imaging sensors (30a - 30e) are of different sensor technology designs

8. The first sensor technology design and the second sensor technology design are selected from the group of sensor technology designs including near-infrared sensors, X-ray sensors, CMYK sensors, RGB sensors, volume measurement sensors, spectroscopic point measurement systems, visible light spectroscopy, near-infrared spectroscopy, mid-infrared spectroscopy, fluorescent X-ray sensors, electromagnetic sensors, laser sensors, multi-spectral systems using LEDs / pulsed LEDs / lasers, laser-induced breakdown spectroscopy (LIBS), fluorescence detection, detectors for visible or invisible markers, transmission spectroscopy, transflectance / intreractance spectroscopy, softness measurement, and thermal cameras, and / or the first sensor technology design and the second sensor technology design are of the same general sensor technology design but have different qualitative differences. The method (100) according to any one of claims 1 - 7

9. One or more input imaging sensors (30, 30a - 30e) configured to obtain input image data (50*) of the object (A - E) in bulk flow A computer-readable storage medium (10) storing the trained neural network (TNN) trained by the method (100) according to any one of claims 1 - 8 Comprising A neural network implementation sensor system (60)

10. The neural network implemented sensor system (60) according to claim 9, configured to be arranged in the bulk flow distribution system (70) and thereby classify the objects (A-E) in the bulk flow to be distributed.

11. Based on the classification provided by the learned neural network (TNN), one or more sorting units (72) are configured to sort the objects (A-E) in the bulk flow, and the one or more sorting units are configured to detect specific types of objects in the bulk flow based on the classification provided by the learned neural network (TNN), and are respectively arranged in or respectively follow each detection area so configured. The neural network implemented sensor system (60) according to claim 10.

12. The neural network implemented sensor system (60) according to any one of claims 9-11, configured to share the learned neural network (TNN) stored in the computer-readable storage medium (10) with a second neural network implemented sensor system (60).

13. The one or more input imaging sensors (30, 30a-30e) are selected from the group of sensor technology designs including near-infrared sensors, X-ray sensors, CMYK sensors, RGB sensors, volume measurement sensors, spectroscopic point measurement systems, visible light spectroscopy, near-infrared spectroscopy, mid-infrared spectroscopy, fluorescent X-ray sensors, electromagnetic sensors, laser sensors, multi-spectral systems using LEDs / pulsed LEDs / lasers, laser-induced breakdown spectroscopy (LIBS), fluorescence detection, detectors for visible or invisible markers, transmission spectroscopy, transflectance / intreractance spectroscopy, softness measurement, and thermal cameras, and / or at least two of the plurality of input imaging sensors (30, 30a-30e) are of the same general sensor technology design but have different qualitative differences. The neural network implemented sensor system (60) according to any one of claims 9-12.

14. A method (200) for sorting objects (A-E) in a bulk flow by a sorting system (70) comprising a neural network implementation sensor system (60) according to any one of claims 9-13, wherein obtaining input image data (50*) of the objects (A-E) in the bulk flow in a first detection area (50a-50e) by at least one input imaging sensor (30a-30e) of the first sensor technology design (step 201); classifying the objects (A*-E*) depicted in the input image data (50*) using the classification output by the learned neural network (TNN) of the computer-readable storage medium (10) by a processing unit (20) (step 202); sorting the objects (A-E) in the bulk flow by a sorting unit (72) based on how the depicted objects (A*-E*) are classified (step 203); A method (200) comprising the steps of

Images