Material flow

Abstract

The invention discloses a data processing device for a sorting device, a method for sorting materials and a sorting device. The data processing apparatus comprises: a first data processing entity configured to operate a sorting application to process image data from an image data source; and a second data processing entity configured to operate an analysis application using a neural network to process the image data from the image data source, the second data processing entity configured to process image data of a higher resolution than the first data processing entity. An image data resolution converter is provided to reduce a resolution of the image data from an image data source prior to processing by a sorting application of a first data processing entity.

Description

Technical Field

[0001] The present disclosure relates to apparatus and methods for handling material flows, and more particularly to sorting and analyzing materials based on image data of the material flow from optical monitoring equipment. Background Art

[0002] Industrial processes may involve processing a large amount of materials flowing through the process. Continuous monitoring of materials may be required at various stages of the process. Monitoring of materials may be performed by specific optical imaging equipment, which is configured to generate digital image data of materials, which may then be processed by a data processing device. Data signals may be generated by imaging devices (imaging equipment such as digital cameras or scanners, optical readers, etc.). Signals input into the data processing device from the imaging device may generate appropriate process control actions and reports. Monitoring based on data from imaging equipment may be provided for various reasons, may be applied to various types of materials, and may generate various actions. Materials may include bulk materials. Bulk materials are generally understood to be essentially dry materials in powder, granular or block form, and may be processed in large quantities. Non-limiting examples of materials include food substances, such as rice, grains, beans, nuts, fruits, vegetables, etc. Other examples include polymer materials, metal materials and wood materials, mixed materials, waste materials, and any other physical substances that are processed.

[0003] Processing based on image data may include sorting of materials. Sorting is an established operation and encompasses a wide variety of material handling tasks that may be performed based on monitoring materials and the resulting sorting of the materials in some manner. For example, sorting may include tasks such as grading or classifying a product stream and removing unwanted items from a product stream.

[0004] In addition to sorting, various analyses of the material can be provided based on the image data. Information on aspects such as mass, volume, number, particle size, quality, color, type, moisture, etc. can be generated. Information on other characteristic related parameters can also be generated, for example, the amount of damaged items, the type of damage, product variations, etc.

[0005] There are many ways to sort and analyze material streams, and the above examples are not presented as an exhaustive list of sorting and analysis applications.

[0006] Sorting and analysis operations have conflicting requirements. The sorter operates in real time during the material flow and the processing of image data needs to be fast so that the sorter can react. On the other hand, the analysis function needs to provide accurate results. The analysis does not necessarily need to be real time. Due to the different requirements, current systems operate based on separate sorting and analysis functions. Summary of the invention

[0007] According to one aspect, a data processing device for a sorting device is provided, comprising: a first data processing entity, configured to operate a sorting application to process image data from an image data source; a second data processing entity, configured to operate an analysis application using a neural network to process the image data from the image data source, the second data processing entity being configured to process image data with a higher resolution than the first data processing entity; and an image data resolution converter, configured to reduce the resolution of the image data from the image data source before being processed by the sorting application of the first data processing entity.

[0008] According to another aspect, a method for sorting a material flow is provided, comprising: processing image data from an image data source by a first data processing entity configured to provide a sorting application; processing the image data from the image data source by a second data processing entity using a neural network configured to provide an analysis application; wherein the method further comprises: reducing the resolution of the image data from the image data source before processing by the sorting application, processing the reduced resolution image data by the sorting application, and processing the higher resolution image data by the analysis application.

[0009] The source of image data may include a high resolution digital imaging device.

[0010] A memory may be provided for buffering a portion of the image data prior to processing by the second data processing entity.The second data processing entity may be configured to process a subset of the image data.

[0011] According to one aspect, the sorting application and the analysis application can be trained based on the same labeled images.

[0012] The apparatus may be configured to determine differences in classification of the same image data by the sorting application and the analysis application.

[0013] The determined difference may be taken into account in the control of the sorting device. At least one sorting parameter may be optimized based on the determined difference. New training data may be automatically generated for a classifier of the sorting application based on the determined difference.

[0014] The sorting application may be configured to use a first neural network and the analysis application configured to use a second neural network.

[0015] The sorting application and / or the analysis application can be configured to use a convolutional neural network.

[0016] The flow of material can be smoothed by means of a feed-forward type feed tray and a chute with a curved material receiving section. The flow of material can be smoothed in order to even out the flow of material for imaging and / or for sorting equipment based on data processing operations.

[0017] According to one aspect, a sorting device is provided, comprising: a data processing device for generating sorting instructions, a feed-forward type feed tray, and a chute comprising a curved section for receiving material from the feed-forward type feed tray. Separating items in a smooth material flow is based on control instructions from the data processing device.

[0018] A computer software product may be provided to implement the methods and apparatus described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Some aspects will now be described in further detail, by way of example only, with reference to the following examples and drawings, in which:

[0020] Figure 1 A schematic example of a sorting device is illustrated;

[0021] Figure 2 is a schematic example of a possible system architecture according to the principles disclosed herein;

[0022] Figure 3 and Figure 4 A flowchart illustrating operations according to certain examples; and

[0023] Figure 5 Another example of a sorting device is shown. DETAILED DESCRIPTION

[0024] The following description gives an exemplary description of some possibilities for practicing the present invention. Although the specification may refer to "one", "an" or "some" examples or embodiments in several places of the text, this does not necessarily mean that each reference refers to the same (one or more) examples or embodiments, or that a particular feature applies only to a single example or embodiment. Individual features of different examples and embodiments may also be combined to provide other embodiments.

[0025] Figure 1 A side view of a schematic example of a processing system 1 is shown, the processing system 1 being configured to monitor particulate material 4 flowing through the system. The processing system 1 comprises a chute 2 that directs the material 4, as indicated by arrow 3. Note that Figure 1 The schematic diagram of shows the sides of the chute 2 as transparent so that the material 4 is visible, whereas in a practical embodiment the interior of the chute may not be visible. Instead of or in addition to the chute, the processing system may also include a conveyor or other device that affects the flow of material in the system.

[0026] The chute 2 directs the material 4 to a material handling device 5, such as equipment configured to sort the material into different streams or to discharge rejected objects from the material stream. The material handling device can process the material in various ways. For example, the material can undergo quality control actions, where the material can be sorted (such as accepted / rejected) based on the detection of impurities, deformed objects, incorrectly sized objects, discolored objects, damaged objects, changes, etc. Other non-limiting examples include grading or classification of materials, where different grades or types of materials are separated based on control instructions from a control system. For example, food materials or waste materials can be divided into different material streams based on the type of material determined by the material handling device 5.

[0027] The material handling apparatus 5 is controlled via a data link 11 by a controller system including a data processing apparatus 10. The control action is based at least in part on image data from the imaging equipment 6 via a data link 8. The imaging equipment comprises an imaging device, such as a digital camera, configured to image an imaging area 7 of the monitoring station. The term imaging area is used to refer to the field of view of the camera on the imaging plane. In line scanning applications, the imaging area is provided as a substantially narrow strip. This enables imaging of line scanning applications and directly taking images of the imaging area across the end of the chute. The line of sight from the camera to the imaging area is indicated by a dotted line. The imaging or viewing area 7 is typically a narrow strip extending over the width of the free-falling material stream emerging from the chute 2, and the camera is configured to image the entire length and width of the imaging area so as to image all material 4 passing through the imaging area.

[0028] The image captured by the camera's optics is converted into a digital signal for input to a data processing device 10 via a data connection 8. An interface 13 is provided to connect the data processing device to the camera. The interface component may also be used for selective resolution conversion tasks, which will be explained in more detail later.

[0029] The data connections 8 and 11 may be any suitable wired or at least partially wireless data connections and may transmit data based on any suitable data communication protocol. The data processing device 10 may include any suitable computer device, such as a server, configured to perform the sorting, analysis and control functions described herein. The data processing device 10 may be connected via an interface 12 to become part of a wider data communication system, such as an intranet or the Internet.

[0030] The data processing apparatus 10 may include any suitable computing arrangement including at least one processor 14, 15, memory 16, software and internal circuitry and components necessary to perform tasks configured to implement at least some of the features described herein. Figure 1Only examples of internal components of a data processing device arranged on a board and including (one or more) processors and one or more memories and connections between elements of the device are shown, and it should be appreciated that the functionality of the data processing device can be arranged in various ways. The processor can be configured to execute appropriate software code to implement one or more of the following aspects. The software code can be stored in at least one memory. The data processor device can be connected to one or more cloud or virtualized environments. According to one possibility, at least part of the tasks of the data processing device 10 is provided based on cloud computing. Appropriate gateways can be provided between different environments.

[0031] The data processing device can be configured to monitor and control large-scale production facilities, such as factories or food processing plants. For example, the data processing device can be configured for food grinders, various types of food processing applications, recycling plants, different raw material processing facilities, etc.

[0032] Artificial intelligence (AI) can be used to perform sorting and / or analysis functions. According to an example, AI is used in analysis but not in sorting. When a method using artificial intelligence (AI) is used, the AI ​​is implemented using one or more neural networks operating on a modern graphics processing unit (GPU) with reasonable cost and complexity, the speed requirements of the sorting function can be achieved if lower resolution input data is processed, while good quality analysis functions can be achieved with higher resolution input data.

[0033] Currently, sorting and analysis functions are performed by separate processing for sorting and analysis tasks. Sorting must be provided immediately and in real time by nature, while analysis can tolerate a certain delay. However, using two separate equipment will incur unnecessarily high costs, delays in obtaining analytical results, and will cause additional mechanical handling of the product and complexity of the equipment.

[0034] Certain aspects of efficiently sorting products in a bulk material stream and simultaneously performing quality analysis on the products in the bulk material stream are described below. Instead of using separate equipment and stages for sorting and analysis functions, a single stage sorting and analysis process is provided based on the same image data. The image data is processed to adjust the resolution to suit the needs of different applications. The data processing device can be configured to have a first data processing entity, which is configured to operate a sorting application to process image data from an image data source. A second data processing entity is also provided, which is configured to operate an analysis application using a neural network to process the image data from the image data source. The first data processing entity can also be configured to use a neural network. The second data processing entity can be configured to process the image data substantially simultaneously with the first data processing entity.

[0035] The second data processing entity is configured to process image data at a higher resolution than the first data processing entity. This may be achieved using an image data resolution converter configured to reduce the resolution of said image data from the image data source before processing by the first data processing entity providing the sorting application.

[0036] Figure 2 A more detailed example of the architecture of a system 20 according to the principles explained herein is shown. A high-resolution camera 21 provides an image data signal for use by an AI method for object detection and quality analysis using two separate data processing entities. Currently, the typical image resolution for sorting applications is 0.3 mm / pixel. Analytical applications are typically based on using higher resolutions, such as 0.15 mm / pixel or finer. Therefore, the camera 21 may include a camera that generates a higher resolution image data output than is necessary for the sorting application, and the output signal is processed so that it can be used for both sorting applications and analytical applications. In order to provide this, the sorting path may include a resolution reduction step 22 before being input to the processing entity 23. The processing entity 23 processes all input data substantially in real time. According to one possibility, the data processing entity 23 may be configured to provide sorting based on making decisions using AI. A second data processing entity 25 is used in parallel for AI-based analysis. The same camera data output may be used by both the sorting processor entity and the analytical processor entity. The analytical processor entity may output the desired quality data about the monitored material.

[0037] An analysis processor may be provided which is fast enough to process the image data as quickly as the data arrives. In this case, buffering prior to analysis is unnecessary. However, in some scenarios, due to the length of the processing pipeline, a significant lag (delay) may occur in generating the output. To address situations where the analysis processor cannot keep up with the rate at which the image data is being captured, the analysis path may include a memory 24 for buffering the data. The analysis processor 25 may be configured to process only a subset of the image data. In addition to, or as an alternative to, buffering the image data, some of the image data may be ignored.

[0038] The sorting processor entity may be arranged to control a sorting device 27, such as a reject ejector.

[0039] A central controller 26 may be provided for processing decision data from the first and second data processor entities 23, 25. As shown, the controller 26 may signal feedback to at least the first data processor entity 23. Examples of such feedback will be described later in this description.

[0040] Feedback may also be provided to a second data processor entity or another entity. Feedback may be useful for a variety of purposes. For example, feedback from an analysis processor may be used to determine if there is a difference in the sorting output between a sorting processor and an analysis processor. Significant differences may indicate that one of the processors is underperforming.

[0041] The AI ​​functionality of processing entities 23 and 25 may be provided by a convolutional neural network (CNN). A CNN is a deep learning artificial hierarchical neural network. CNNs are often applied to analyze visual imagery and are used in applications such as image and video recognition, recommender systems, image classification, object detection, and image segmentation. CNNs exploit hierarchical patterns in the data and assemble increasingly complex patterns using smaller and simpler patterns embossed in their filters. Compared to other image classification algorithms, CNNs use relatively little preprocessing. The network learns through automatic learning to optimize filters (or kernels), whereas in more traditional algorithms these filters are hand-designed. This independence from prior knowledge and human intervention in feature extraction can provide advantages in applications such as sorting and analysis of bulk material streams.

[0042] Figure 3 A flow chart of a method for sorting equipment is shown, wherein high resolution image data is received from a high resolution image data source (e.g., a digital camera or a line scanner) at 100. The resolution of the image data is reduced at 102 and then processed at 104 by a first data processing entity configured to provide a sorting application. Simultaneously, the high resolution image data is processed at 106 using a neural network of a second data processing entity configured to provide an analysis application. The processing at 106 may be separated into subsets of the image data. The results of the processing are output at 108.

[0043] Figure 4 A flow chart of an arrangement for configuring two processing entities to provide a neural network is shown. High resolution image data is received from a high resolution image data source at 200. The resolution of the image data is reduced at 202 and then processed at 204 by a first data processing entity configured with a first neural network to provide a sorting application. Simultaneously, the same high resolution image data is processed at 206 using a second neural network of a second data processing entity configured to provide an analysis application. The result of the processing is output at 208.

[0044] The principles explained herein can be particularly useful when the data processing for both the sorter and the analyzer uses a convolutional neural network (CNN). This can be because the classifiers for the sorter and the analyzer can be trained from the same labeled images. This can be advantageous because the step of creating labeled data to train the sorter can be a limiting factor in creating new sorting and / or analysis applications. It is expected that future sorting applications will have more powerful processing capabilities and are therefore expected to be suitable for quickly handling high-resolution image data in essentially real time. When these become available, existing labeled data can also be used for these high-resolution sorting applications, and the need for manual intensive retraining of sorting AI applications can be reduced or completely eliminated.

[0045] The method also facilitates comparison of results from the sorting processor and the analytical processor, since an inherent correspondence already exists between the two inherently different processing applications. Figure 2 The difference between the classification decisions of the two independent processing entities 23 and 25 can be used as feedback data to control and optimize the parameters of the sorter. Figure 2 In the example above, this is represented by feedback to the central controller 26. This can even be provided as a run-time operation based on feedback data from the analyzer processing entity to the sorting processor entity. Another example is to use the difference between the decisions of the two processors to automatically generate new training data for the classifier of the sorting processor.

[0046] Figure 5 An example of a sorting device is shown which includes a forward feed tray 31 for feeding material 34 onto a chute 32. The material receiving portion of the top of the chute is provided with a curved chute section 33 for smoothing the flow of material 34 fed to the chute. In operation, the forward feed tray 31 smoothly delivers the material directly to the top of the chute without any reversal of direction, and the curved chute then smoothly transitions the direction of travel from horizontal to inclined.

[0047] The sorting device also includes equipment for removing or separating unwanted material 35 from the flow. The separation equipment includes a separation or ejection actuator 41 and a cooperating separation plate 37. The actuator 41 is controlled by the data processing device 30. The data processing device is configured to generate a control signal for the actuator 41 based on digital image data from the imaging system 40, as explained above. The actuator may be pneumatic, so that in operation it blows air at the rejected items 35 so that the travel path of the rejected items is changed to guide them to the reject side of the edge of the separation plate 37. By means of this, the items 35 determined by the data processing device 30 to be rejected can be separated from the flow 34. The acceptable material items 36 are not affected and can continue their travel path to the next processing stage.

[0048] Smoothing the flow of material 34 by the feed-forward tray 31 and the curved flow-smoothing section 33 can provide the imaging system 40 with better visibility of the material flowing through the imaging area. In addition, the smoothed flow makes it easier for the actuator 41 to correctly eject items 35 determined to be rejected from the flow.

[0049] For example, such a feeding mechanism can be advantageous when the sorting processor includes a convolutional neural network (CNN) implemented on a graphics processing unit (GPU). A "smoother" feed can be particularly useful for CNN-based sorting, which typically has greater latency than non-CNN-based methods. Greater latency can require a larger gap between the line of sight of the imaging device and the line where the actuator 41 performs sorting based on the image data. Efficient sorting with a larger gap can benefit from a smoother feed to reduce sorting errors due to an increased likelihood of good products falling into a reject channel or an ejector missing a defect.

[0050] Figure 5 Also shown is an imaging system 40 comprising more than one camera. In this example, dual sided viewing is provided by two cameras 38 and 39 after the material stream 34 leaves the end of the chute 32. It is also possible to provide a single sided viewing imaging arrangement on the chute, where the material is imaged from one side.

[0051] Imaging can be provided while the material is on the chute or after the material has left the chute.

[0052] The principles described herein provide a sorting device that is capable of sorting products and analyzing the quality of products flowing through the sorting equipment substantially simultaneously. The advantage is that accurate analysis can be performed while potentially providing a low-cost and low-complexity sorting implementation without compromising the efficiency, yield, and throughput of the sorting equipment. Moreover, future upgraded training of AI-based sorting applications can be more efficient. For example, an application can be upgraded from a 0.3 mm pixel resolution application to a 0.15 mm pixel resolution sorting application based on labeled image data without retraining or with minimal retraining.

[0053] It is noted that, although the above detailed examples are described with reference to certain processes, applications and machines, several variations and modifications may be made to the disclosed solutions without departing from the scope of the invention. In particular, different embodiments have been described as examples. Different features from different embodiments may be combined.

[0054] The above description provides a complete and informative description of exemplary embodiments of the present invention by way of exemplary and non-limiting examples. However, various modifications and adaptations may become apparent to those skilled in the relevant art when the above description is read in conjunction with the accompanying drawings and the appended claims. All such and similar modifications to the teachings of the present invention will still fall within the spirit and scope of the present invention.

Claims

1. A data processing device for a sorting device, comprising: a first data processing entity configured to operate a sorting application to process image data from an image data source; a second data processing entity configured to operate an analysis application using a neural network to process said image data from said image data source, said second data processing entity being configured to process higher resolution image data than said first data processing entity; as well as An image data resolution converter is configured to reduce the resolution of the image data from the image data source before being processed by the sorting application of the first data processing entity.

2. The data processing apparatus according to claim 1, wherein the image data source comprises a high-resolution digital imaging device.

3. The data processing apparatus according to claim 1 or 2, further comprising a memory for buffering a portion of the image data before processing by the second data processing entity, the second data processing entity being configured to process a subset of the image data.

4. A data processing apparatus according to any of the preceding claims, wherein the sorting application and the analysis application are trained based on the same labeled images.

5. The data processing apparatus according to any of the preceding claims, configured to determine differences in classifications of the same image data by the sorting application and the analysis application.

6. The data processing apparatus according to claim 5, configured to do at least one of the following: taking the determined differences into account in the control of the sorting device, optimizing at least one sorting parameter based on the determined difference, and New training data is automatically generated for the sorting application based on the determined differences.

7. A data processing apparatus according to any one of the preceding claims, wherein the sorting application is configured to use a first neural network and the analysis application is configured to use a second neural network.

8. A data processing apparatus according to any one of the preceding claims, wherein the sorting application and / or the analysis application is configured to use a convolutional neural network.

9. A method for sorting a material stream, comprising: processing image data from an image data source by a first data processing entity configured to provide a sorting application; processing the image data from the image data source by a second data processing entity using a neural network configured to provide an analysis application, wherein the method further comprises: reducing the resolution of the image data from the image data source prior to processing by the sorting application; processing the reduced resolution image data by the sorting application; and The higher resolution image data is processed by the analysis application.

10. The method of claim 9, comprising receiving the image data from a high resolution digital imaging device.

11. A method according to claim 9 or 10, comprising buffering a portion of the image data prior to processing by the neural network of the analysis application, and processing the buffered image data in a subset by the neural network.

12. The method according to any one of claims 9 to 11, wherein the sorting application is configured to use a neural network for sorting, the method comprising training the neural network of the sorting application and the neural network of the analysis application based on the same labeled images.

13. The method according to any one of claims 9 to 12, comprising determining differences in classifications of the sorting application and the analytical application based on the same image data.

14. The method according to claim 13, comprising at least one of the following: Use the determined differences in controlling the sorting equipment, optimizing at least one sorting parameter based on the determined difference, and New training data is automatically generated for the sorting application based on the determined differences.

15. A sorting device comprising: The data processing device according to any one of claims 1 to 8; a chute including a curved material receiving section; a forward feed tray for feeding material to the curved material receiving section of the chute; as well as Separation equipment is used to separate items in the material based on control instructions from the data processing device.

Images