Selection by automated learning

Abstract

The present invention discloses a method for controlling the sorting of items in a plurality of sorting configurations, and a computer-readable storage medium storing instructions for implementing the method. The method includes, in a first sorting configuration (1), sorting an item flow (2) into at least one sorted section (3) and optionally at least one unsorted section (4) (S1), capturing a set of section images (S2), labeling the section images with section labels (S3), and constructing at least one classifier configured to associate items with sections (S4), wherein at least one classifier is a machine learning model. Furthermore, in a second sorting configuration (10), capturing images of items in an item flow (12) (S5), and using the images captured in the second sorting configuration (10) and at least one classifier to control the sorting of items in the item flow (12) into at least one sorted section (13) and optionally at least one unsorted section (14) (S6).

Description

Technical Field

[0001] The concept of the present invention generally relates to a method for controlling the sorting of materials in a plurality of sorting arrangements.

Background Art

[0002] Existing sorting arrangements sort materials into different categories. The sorting process can include manual, physical, and optical sorting steps, as well as combinations of these steps. For example, in the sorting of fibrous materials, the flow of clothing can be manually sorted into marketable categories such as "fashionable jeans".

[0003] Such processes, especially those involving more complex selections, are often specially adjusted and optimized according to the on-site situation. That is, it takes time and cost to properly set up a new sorting arrangement.

[0004] Therefore, there is a need for an improved method for setting up a new sorting arrangement more efficiently in terms of time and cost.

Summary of the Invention

[0005] In order to achieve at least one of the above objects and other objects apparent from the following description, according to the concept of the present invention, a method having the features defined in claim 1 is provided. Preferred variations will be apparent from the dependent claims.

[0006] According to a first aspect of the present invention, a method is provided for controlling the sorting of items in a plurality of sorting arrangements, the method comprising: sorting the flow of items in a first sorting arrangement into at least one sorted section and optionally at least one unsorted section; capturing a set of section images of items within at least one sorted section; labeling the section images captured in the first sorting arrangement with section labels, each section label indicating the sorted section within the at least one sorted section into which the depicted item was sorted, and associating the item with the section. The present invention relates to constructing at least one classifier configured such that at least one classifier is a machine learning model, and constructing at least one classifier involves training a machine learning model using segmental images and segmental labels captured in a first sorting setup, and in a second sorting setup, capturing images of items in the item flow, and using the images captured in the second sorting setup and at least one classifier, controlling the sorting of items in the item flow to at least one sorted segment and optionally at least one unsorted segment.

[0007] This provides a method for automating and simplifying the sorting process in multiple sorting configurations. Since the classifier does not require information about the internal workings of the actual sorting process and is constructed solely by observing at least one sorted section, a more cost-effective and time-efficient solution is provided.

[0008] This method can be used in all kinds of sorting processes. That is, this method can be implemented in all kinds of sorting arrangements and all kinds of item flows. Thus, the item flow can include all kinds of items or any combination of items, such as clothing, minerals, foodstuffs, empty containers such as metal cans, etc. That is, the items in the item flow can be made of or contain plastics, metals, textile materials, organic materials, mineral materials, etc.

[0009] In the context of this disclosure, the term “sorting arrangement” should be understood as a system located in a specific place on which a sorting process is performed. A sorting arrangement may include, for example, a conveyor belt that carries a flow of items, on which operators performing a manual sorting process are located. Alternatively, a sorting arrangement may include a conveyor belt that carries a flow of items, which is monitored by an automated sorting mechanism. Thus, the first and second sorting arrangements should be understood as spatially separated, i.e., sorting different flow of items. The first sorting arrangement may be located in a first sorting plant, while the second sorting arrangement may be located in a second sorting plant. Alternatively, the first and second sorting arrangements may be located in the same sorting plant. Thus, the method can be implemented in sorting arrangements located in different locations in a plant, different locations in a city, different locations in a country, different countries, and / or different continents.

[0010] In the first and / or second sorting configuration, each item flow may be sorted into one or more sorted categories. Each item flow may, for example, be sorted into the first and second sorted categories. Alternatively, an item flow may be sorted into a different number of sorted categories. That is, an item flow in the first sorting configuration may be sorted into two sorted categories, while an item flow in the second sorting configuration may be sorted into one sorted category. In other words, even if a classifier is constructed using two sorted categories, the classifier may be distributed and implemented in another sorting configuration to sort an item flow into only one sorted category.

[0011] In the context of this disclosure, the term “unsorted category” should be understood as a category whose contents have not been intentionally selected. That is, at least one unsorted category contains items that have not been sorted into at least one sorted category. The item flow may be clothing that includes both trousers and shirts of different colors, and the sorted category consists of red shirts. As a result, the unsorted category contains shirts of colors other than red, and trousers of different colors. Thus, such an unsorted category may constitute items that should be removed from the sorting process. Additionally or alternatively, the unsorted category may be recycled within a sorting arrangement or provided to another sorting arrangement.

[0012] In an exemplary sorting arrangement, the item flow is provided in the form of a flow of garments including trousers of different colors, where trousers of a first color are sorted into a first sorted section, and trousers of a second color are sorted into a second sorted section. Optionally, garments of other colors besides the first and second colors are sorted into an unsorted section. For example, such an unsorted section may be provided to another sorting arrangement for sorting into a sorted section.

[0013] In another exemplary sorting arrangement, the flow of items is provided in the form of flows of different types of clothing, with trousers sorted into a first sorted section and shirts sorted into a second sorted section. Optionally, other types of clothing besides trousers and shirts may be sorted into an unsorted section. For example, such an unsorted section may be provided to another sorting arrangement for sorting into a sorted section.

[0014] In another exemplary sorting arrangement, the flow of items is provided in the form of different types of clothing, with trousers sorted into a first sorted section and shirts into a second sorted section. Optionally, objects made of other materials may be sorted into an unsorted section. For example, such an unsorted section may be provided to another sorting arrangement for sorting into a sorted section, or it may be completely disposed of, for example, by dumping or incineration.

[0015] In the context of this disclosure, the term "depicted" for an item should not be interpreted as meaning that the item must be imaged by detecting electromagnetic waves within the visible wavelength band. In other words, "depicting" an item can be done with electromagnetic waves of any wavelength, including infrared, ultraviolet, and X-rays.

[0016] Taking a set of section images of items within at least one sorted section can be done at any point after the items have been separated from the item flow. For example, if the sorting setup includes a conveyor belt that provides a flow of items to a location where physical sorting takes place, and after that physical sorting process each sorted section is transported on another conveyor belt, the section images may be taken at any moment when the items in the sorted section are on that other conveyor belt. As another example, the section images may be taken when the items in the sorted section are stationary after they have been separated from the item flow.

[0017] A set of segmented images should be understood as being able to contain any number of non-zero images. For example, a set of segmented images can contain one or more images, such as 10 images, or 100 images, or 10000 images, or 10000 images.

[0018] For any single item, two or more segmented images may be captured. As a result, one item may exist in two or more images. Furthermore, two or more items may exist in any single figure. For the purposes of the concept of this invention, it is not necessary to be able to track any single object. Rather, the purpose of the set of segmented images is to train a classifier, i.e., a machine learning model.

[0019] The image may be captured by any type of camera configuration known in the art.

[0020] The machine learning model may be any type of machine learning model. For example, the machine learning model may be a neural network such as a convolutional neural network or a fully connected neural network. The machine learning model may be a neural network having any number of input nodes, any number of hidden layers and each node within those layers, and any number of output nodes. Furthermore, any node in the neural network may have any type of activation function, such as a rectified linear unit (ReLU) activation function or a logistic function.

[0021] As is well known in the art, an image label should be understood as information attached to or associated with an image that can be important for training a machine learning model, such as any features depicted by the image or any context in which the image was taken. Here, a label indicates a selected category from at least one selected category to which the depicted item was selected. That is, a label may be a keyword such as "green," or "trousers," or "aluminum." More than one label may be attached to or associated with a single image. A label may be a multiclass label, i.e., a label containing two or more elements. That is, a label may be a vector containing multiple elements. For example, a label may be a multiclass label containing the elements "trousers," "short," and "green."

[0022] Labeling segmented images further involves identifying at least one depicted item for each segmented image, and the segmented label further indicates each identified item.

[0023] Identifying a depicted item may include, for example, determining the category or type of the item. For instance, identifying a depicted item may include determining that the item is of a type such as "trousers," or "shirt," or "red trousers," or "blue shirt," or "can," or "aluminum can." Furthermore, segmented images should be understood as being able to depict two or more items. Therefore, labeling a segmented image may involve identifying two or more items for any given image.

[0024] Identifying depicted items can be done in various ways. For example, depicting items may be done through foreground-background segmentation, or through object instance segmentation.

[0025] Object instance segmentation can be used to distinguish individual items within an overlapping list of items, making it particularly useful when any items overlap within a flow of items.

[0026] Object instance segmentation may be performed by a machine learning model. For example, the machine learning model may be part of the classifier or the same machine learning model as the classifier. Therefore, object instance segmentation may be performed by a neural network, such as a convolutional neural network or a fully connected neural network. Images may be provided to the classifier as training material after being processed by a machine learning model using object instance segmentation.

[0027] The images in the set of classified images can be the subject of further image processing steps. For example, the images in the set of classified images can be the subject of image improvement, cropping, etc. Furthermore, the captured classified images should be understood to be able to be discarded from the set of classified images, for example, when an object cannot be identified at all or when the image quality is too low.

[0028] This method may further include extracting a set of classified item images from the set of classified images, such that each classified item image depicts one identified item, so that the number of classified item images in the set of classified item images can be the same as the number of items identified within the set of classified images, and labeling is performed on the set of classified item images, and the training of the machine learning model can be performed using the classified item images and classification labels captured in the first sorting arrangement.

[0029] Thereby, when an image depicts two or more items, a set of item images may be extracted from the image, thereby generating a clearer training dataset. Also, the classified item images can have a smaller data size than the original classified images. Thereby, a set of images that are easy to process is provided.

[0030] It should be noted that any one of the item images in the set of classified item images can be discarded.

[0031] This method may further include, in the first sorting arrangement, capturing a set of stream images of the items in the item stream before sorting the item stream into at least one sorted category and optionally at least one unsorted category, and labeling the stream images with stream labels, and labeling may further include identifying at least one depicted item for each stream image, and the stream labels may further indicate each identified item.

[0032] A stream of images should be understood as being able to contain any number of non-zero images. For example, a stream of images can contain one or more images, such as 10 images, or 100 images, or 1000 images, or 10000 images.

[0033] A set of stream images may be processed as described above for a set of segmented images. Specifically, images within a set of stream images may be subject to further image processing steps. For example, images within a set of stream images may be subject to image enhancement, cropping, etc. Furthermore, it should be understood that captured stream images may be discarded from a set of stream images, for example, if an object is not identifiable at all, or if the image quality is too low.

[0034] Since a stream image may depict two or more labels, or a multi-class label, for a stream image, it may represent two or more items that are substantially different from each other from a classification standpoint, namely, "blue jeans" and a "green shirt," or "aluminum can" and a "plastic bottle."

[0035] Labeling stream images may further involve extracting a set of stream item images from a set of stream images, where each stream item image depicts one identified item, such that the number of stream item images in the set of stream item images may be equal to the number of items identified in the set of stream images, and the labeling may be performed on the set of stream item images.

[0036] A set of stream images or a set of stream item images can be used in a variety of ways.

[0037] The method may further include comparing a set of stream images with a set of segmented images, or comparing a set of stream item images with a set of segmented item images, removing images from the set of stream images or stream item images that depict items depicted in either one of the segmented images or segmented item images to generate a modified set of stream images or modified set of stream item images, and labeling each stream image in the modified set of stream images or each stream item image in the modified set of stream item images with a label indicating that the depicted item has been sorted into at least one unsorted segment, and constructing at least one classifier may further include training a machine learning model using the stream images in the modified set of stream images or stream item images in the modified set of stream item images taken in the first sorting setup, and the labels associated with each.

[0038] This allows machine learning models to be trained on larger datasets and not only to distinguish items that should be sorted into pre-sorted categories, but also to distinguish items that should not be sorted into such categories. Thus, a more accurate classifier is provided.

[0039] Labeling segmented images can be done in various ways.

[0040] Preferably, the labeling of the segmented images captured in the first sorting arrangement with segmentation labels is performed by the processing unit, and the processing unit is further configured to provide the segmented images captured in the first sorting arrangement and the segmentation labels to a machine learning model.

[0041] It should be noted that the above also applies to labeling any other set of images, such as a set of stream images or a set of modified stream images. In other words, labeling stream images or modified stream images may be performed by the same processing unit that labels segmented images.

[0042] The processing unit may be a central processing unit. Alternatively, the processing unit may include multiple components that are isolated from each other and connected in a communicative manner. The processing unit may further be configured to perform the functions of collecting and compiling a set of segmented images and / or a set of streamed images, comparing the set of streamed images with the set of segmented images, removing images from the set of streamed images that depict items depicted in any one of the segmented images, and thereby generating a modified set of streamed images. Furthermore, the processing unit may be configured to provide a machine learning model with the streamed images in the modified set of streamed images captured in the first sorting arrangement, along with their respective associated labels. Furthermore, the processing unit may be configured to perform the function of rejecting outlier images, i.e., images that deviate from the rest of the set of images by a predetermined tolerance level. The rejection of outlier images may optionally be supplemented by manual intervention by an operator.

[0043] The classifiers can be distributed to the second sorting configuration in various ways. For example, the classifiers may be stored in a non-temporary storage medium in the first sorting configuration, and that medium may then be transported to the second sorting configuration.

[0044] Preferably, the method further includes distributing at least one classifier from a first sorting configuration to a second sorting configuration via a network infrastructure such as a cloud-based server.

[0045] This allows for the automation of the distribution of at least one classifier.

[0046] In the second sorting configuration, the classifier may be implemented in, for example, a control unit. Such a control unit may be configured to communicate with the first sorting configuration via the same network infrastructure, for example, by communicating with the same cloud-based server.

[0047] The method may further include updating at least one classifier distributed to a second sorting configuration, which includes repeatedly constructing at least one classifier in the first sorting configuration by sorting the flow of items into at least one sorted configuration and optionally at least one unsorted configuration, capturing a set of configuration images of items in at least one sorted configuration, labeling the configuration images captured in the first sorting configuration with configuration labels, and training a machine learning model using the configuration images and configuration labels captured in the first sorting configuration, thereby providing at least one updated classifier, distributing the at least one updated classifier to the second sorting configuration, and activating the at least one updated classifier in the second sorting configuration.

[0048] This process first updates the classifier in the first selection configuration by further training the machine learning model, and then distributes the updated classifier to further selection configurations. Since the classifier is centrally managed and only needs to be updated in the first selection configuration, a more efficient method is provided.

[0049] The activation of at least one updated classifier in a second sorting configuration may involve, for example, the deactivation of at least one previous version of the classifier, i.e., the updated classifier simply replacing the previous classifier at a functional level, i.e., as the classifier controlling the sorting process. This allows the previous classifier to be retained in the system and selectively activated as needed. Thus, a more accurate classifier is provided.

[0050] Updating at least one classifier distributed to the second sorting configuration may further include sending a validation request from the second sorting configuration to the first sorting configuration, if the validation request fails, the first sorting configuration may repeat the process of constructing at least one classifier, which includes sorting the flow of items into at least one sorted section and optionally at least one unsorted section, taking a set of section images of items in at least one sorted section, labeling the section images taken at the first sorting configuration with section labels, and training a machine learning model using the section images and section labels taken at the first sorting configuration, thereby providing at least one updated classifier, distributing at least one updated classifier to the second sorting configuration, and activating at least one updated classifier at the second sorting configuration.

[0051] The verification request may include requesting a comparison between the machine model version of the first sorting configuration and the machine model version of the second sorting configuration. Therefore, a verification request failure may occur if it is determined that the versions are not the same.

[0052] Alternatively, the verification request may be sent from the first sorting configuration to any further sorting configuration. In other words, updates from the first sorting configuration may be carried over to further sorting configurations.

[0053] Training a machine learning model may further include embedding data of segmented images taken in a first sorting configuration and associated segment labels, and optionally data of stream images taken in a first sorting configuration and associated stream labels, as data points in a spatial map such as a two-dimensional or three-dimensional map, and dividing the spatial map into cells, each cell having a data point density and a last input date and time indicating the date and time when the most recent data point was added to the cell.

[0054] The data point density of a cell can also be called the embedding density. Embedding data from an image set can be useful for a variety of reasons. For example, repeating the process of building at least one classifier may further involve, for each cell, checking the data point density and / or the last input date and time, and if the cell's data density is above a certain threshold and / or the cell's last input date and time is the same as or older than the certain threshold, modifying the set of data points contained within the cell, for example, by deleting all data points from the set of data points.

[0055] In other words, if the density of data points in a cell is too high, or if the last entry date and time are too old, the set of data points contained in the cell may be modified accordingly. For example, if the last entry date and time are too old, i.e., too much time has passed since the data points were added to the cell, all data points in the cell may be deleted. This may be useful, for example, if the data points in the cell represent items that are no longer suitable for selection.

[0056] Additionally or alternatively, repeating the process of building at least one classifier may further include, when adding a new data point to the spatial map, checking the data point density and / or last input date and time of the cell to which the data point is being added, discarding the new data point if the cell's data density is above a predetermined threshold, and / or adding the new data point and removing at least one neighboring data point in the cell if the cell's last input date and time is the same as or older than a predetermined threshold.

[0057] This means that when the training dataset is expanded, redundant or outdated data points may be discarded. Therefore, at least one classifier may be updated to have control over the selection of new items, or to have control over the selection of items in a new way.

[0058] The predetermined threshold value for the final input date and time may be, for example, one week, one month, multiple months, one year, or multiple years.

[0059] The predetermined threshold for data density may be a numerical value, for example, each numerical value assigned to a cell, such as 10, 20, or 100, indicating that a cell can only contain 10, 20, or 100 data points. All cells may have the same threshold. Cells may have different thresholds than other cells.

[0060] Alternatively, a predetermined threshold for a cell may be set according to a specified ratio, which is calculated by dividing the total number of data points in all cells by the number of cells. That is, a predetermined threshold for a cell could be 10 data points per cell, 20 data points per cell, or 100 data points per cell. Therefore, the exact value may vary depending on the total number of data points the dataset contained at the time the predetermined threshold was set. Furthermore, the predetermined threshold may change over time. The predetermined threshold may be refreshed, for example, each time at least one classifier is updated, providing each cell with an updated ratio.

[0061] Alternatively, the predetermined threshold may be set according to the probability distribution of data points on the spatial map. That is, cells to which data points are frequently added, or cells that already contain many data points, may have a higher predetermined threshold than cells to which data points are not added very frequently, or cells with few data points. This allows all other cells, or at least all other cells with a higher probability density value, to increase their predetermined threshold when a new data point is provided to a cell that has been assigned a low probability density value.

[0062] Additionally or alternatively, particularly when updating at least one classifier, the dataset may be cleaned up before additional data points are added. For example, the dataset may be cleaned up by a certain percentage per cell. For instance, 5%, 10%, 20%, or 30% of all data points in each cell may be removed before any additional data points can be provided to any cell.

[0063] Any number of classifiers may be constructed. For example, the method according to any one of the above claims, in which the construction of at least one classifier is a method comprising constructing a classifier for each sorted section of at least one sorted section. Furthermore, only one classifier may be constructed, configured in the second sorting setup to sort the flow of items into one or more sorted sections. That is, one classifier is configured to control the sorting of the flow of items into multiple sorted sections. Alternatively, two or more classifiers may be constructed. For example, one classifier may be constructed for each sorted section, in which case the second sorting setup controls the sorting of the flow of items into multiple sorted sections by multiple classifiers. For example, two classifiers may be constructed in the first sorting setup, in which case the second sorting setup controls the sorting of the flow of items into their respective sorted sections.

[0064] Any number of classifiers may be distributed to further selection configurations. That is, the number of classifiers constructed in the first selection configuration does not have to be equal to the number of classifiers distributed to a particular selection configuration. For example, two classifiers may be constructed in the first selection configuration, and only one classifier may be distributed to the second selection configuration.

[0065] Furthermore, the classifiers may be distributed to two or more further sorting configurations. Moreover, the number of classifiers distributed to the second sorting configuration may differ from the number of classifiers distributed to the third sorting configuration.

[0066] In other words, the method may further include taking images of items in the item flow in a third sorting setup, and using the images taken in the third sorting setup and a classifier to control the sorting of items in the item flow into at least one sorted category and optionally at least one unsorted category.

[0067] In any further sorting configuration, i.e., a second or third sorting configuration, the sorting process may be configured to take into account the classifiers distributed to such configurations. In other words, the further sorting configurations may be constructed from the outset based on the classifiers.

[0068] According to a second aspect of the concept of the present invention, a computer-readable storage medium is provided which stores instructions for implementing a method according to the first aspect of the present invention.

[0069] Any advantages or technical effects discussed in relation to the first aspect of the concept of the present invention are also applicable to the second aspect of the concept of the present invention and will not be mentioned again to avoid unnecessary repetition. [Brief explanation of the drawing]

[0070] The concept of the present invention will be described in more detail below with reference to the attached drawings. [Figure 1] Figure 1 is a schematic diagram of a system implementing a method according to a first aspect of the concept of the present invention, showing a first sorting arrangement, the first sorting arrangement being communicatively connected to a processing unit, and a second sorting arrangement including a control unit communicatively connected to the processing unit of the first sorting arrangement. [Figure 2] Figure 2 is a flowchart illustrating a method according to a first aspect of the concept of the present invention. [Figure 3] Figures 3A and 3B are schematic diagrams of the spatial embedding of the data. [Figure 4] Figure 4 is a perspective view of an exemplary second sorting arrangement for sorting fibrous materials. [Modes for carrying out the invention]

[0071] The concept of the present invention will be described more fully below with reference to the accompanying drawings, which illustrate preferred modifications of the concept of the present invention. However, the concept of the present invention can be modified in many different ways and should not be construed as being limited to the modifications described herein. Rather, these modifications are provided to ensure that this disclosure is thorough and complete and to fully convey the scope of the concept of the present invention to those skilled in the art. In the drawings, similar numbers refer to similar elements.

[0072] Figure 1 schematically shows the sorting process in the first sorting setup 1. Here, the input flow of item 2 is supplied to the first sorting setup 1.

[0073] Sorting configuration 1 can implement manual sorting, automatic sorting, or any combination thereof. For example, sorting configuration 1 may include a conveyor belt that carries the input flow of item 2, on which an operator is positioned to perform the manual sorting process. Alternatively, sorting configuration 1 may include a conveyor belt that carries the input flow of item 2, which is monitored by an automatic sorting mechanism. An exemplary sorting configuration is shown in Figure 4 and will be discussed in relation to Figure 4.

[0074] The Item 2 flow may include any type of item or any combination of items, such as clothing, minerals, foodstuffs, and empty containers like metal cans. That is, items in the Item 2 flow may be made of or contain plastics, metals, textiles, organic materials, mineral materials, etc.

[0075] In the first sorting configuration 1, the flow of item 2 is sorted into at least one sorted section 3. Here, the flow of item 2 is sorted into the first and second sorted sections 3. However, the flow of item 2 may be sorted into any number of sorted sections, i.e., sorted from 1 to N sorted sections where N is any number. As shown in Figure 1, the flow of item 2 is also optionally sorted into an unsorted section 4. The unsorted section 4 may consist of items to be removed from the sorting process. Additionally or alternatively, the unsorted section 4 may be recycled within sorting configuration 1 or provided to other sorting configurations.

[0076] In exemplary sorting arrangement 1, the flow of item 2 is provided in the form of a flow of clothing including trousers of different colors, with trousers of the first color being sorted into the first sorted section 3, and trousers of the second color being sorted into the second sorted section 3. Optionally, clothing of other colors other than the first and second colors is sorted into the unsorted section 4.

[0077] In another exemplary sorting arrangement 1, the flow of item 2 is provided in the form of different types of clothing, with trousers sorted into a first sorted section 3 and shirts sorted into a second sorted section 3. Optionally, other types of clothing besides trousers and shirts may be sorted into an unsorted section 4.

[0078] In yet another exemplary sorting arrangement 1, the flow of item 2 is provided as objects made of different materials, with plastic objects sorted into a first sorted section 3 and metal objects sorted into a second sorted section 3. Optionally, objects made of other materials may be sorted into an unsorted section 4.

[0079] As further shown in Figure 1, camera configuration 5 observes two sorted sections 3. Camera configuration 5 may include any camera known in the art. Camera configuration 5 is configured to capture a set of sorted images of items in the first and second sorted sections 3. The sorted images can be captured at any point after the items have been separated from the input flow of items 2. For example, if sorting configuration 1 includes a conveyor belt that provides a flow of items 2 to a location where physical sorting takes place, and after the physical sorting process each sorted section is transported on another conveyor belt, the sorted images may be captured at any moment when the items in the sorted section are on that other conveyor belt. As another example, the sorted images may be captured when the items in sorted section 3 are stationary after being separated from the flow of items.

[0080] A set of segmented images should be understood as being able to contain any number of non-zero images. For example, a set of segmented images can contain one or more images, such as 10 images, or 100 images, or 1000 images, or 10000 images. Two or more segmented images may be taken for any single item. As a result, one item may exist in two or more images. Furthermore, two or more items may exist in any single figure.

[0081] This method may further include extracting a set of segmented item images from a set of segmented images. Each segmented item image can depict one identified item such that the number of segmented item images in the set of segmented item images is equal to the number of items identified in the set of segmented item images. For example, three item images may be extracted from an image depicting three items. Labeling may be performed on the set of segmented item images, and the machine learning model may be trained using the segmented item images and segmented labels captured in the first sorting arrangement. This allows for the extraction of an item image set from an image if the image depicts two or more items, thereby generating a clearer training dataset. Additionally, segmented item images may have a smaller data size than the original segmented images. This provides a set of images that is easier to process.

[0082] A set of segmented images or a set of segmented item images is provided to the processing unit 6. The processing unit 6 may be a central processing unit. Alternatively, the processing unit 6 may include a plurality of components that are separated from each other and connected communicatively. The processing unit 6 is configured to be communicatively connected to a camera arrangement 5 that observes two selected segments 3. Therefore, the processing unit 6 may be further configured to control the camera arrangement 5 and / or to acquire data such as a set of segmented images from the camera arrangement 5.

[0083] Furthermore, the processing unit 6 is configured to label the segmented images captured in the first sorting arrangement 1 with segmented labels. The segmented labels indicate the sorted segment 3 into which the depicted items belong. That is, the labels may be keywords such as "green," "trousers," or "aluminum."

[0084] The processing unit 6 implements a machine learning model. The machine learning model may be of any type. For example, the machine learning model may be a neural network such as a convolutional neural network or a fully connected neural network. The processing unit 6 is further configured to provide the machine learning model with the segmented images and segment labels captured in the first sorting arrangement 1. This configures the processing unit 6 to construct at least one classifier configured to associate items with segmentations. Constructing at least one classifier involves training the machine learning model using the segmented images and segment labels captured in the first sorting arrangement 1.

[0085] A further camera configuration 7 observes the flow of item 2. The further camera configuration 7 is configured to capture a set of stream images of items in the flow of item 2 before sorting the flow of item 2 into two sorted sections 3 and optionally an unsorted section 4. The set of stream images should be understood to be able to contain any number of non-zero images. For example, the set of stream images could contain one or more images, such as 10 images, or 100 images, or 1000 images.

[0086] Here, the processing unit 6 is configured to further compare the set of stream images with the set of segmented images, remove the images from the set of stream images that depict items depicted in any one of the segmented images, and thereby generate a modified set of stream images. Furthermore, the processing unit 6 is configured to label each stream image in the modified set of stream images with a label indicating that the depicted item has been sorted into the unsorted segment 4. Furthermore, constructing at least one classifier further includes training a machine learning model using the stream images in the modified set of stream images taken in the first sorting setup and their respective associated labels. That is, the processing unit 6 is further configured to provide the modified set of stream images to a machine learning model for constructing a classifier.

[0087] Please note that the above process of creating a set of corrected images by comparing and deleting images, and the subsequent process of labeling each image in that set of corrected images, can also be applied to sets of item images, such as sets of category item images or sets of stream item images.

[0088] The processing unit 6 may also be configured to identify at least one drawn item for each image, in which case the label associated with the image will indicate the further identified item. Identifying a drawn item may include, for example, determining the category or type of the item. For example, identifying a drawn item may include determining that the item is of a type such as "trousers," or "shirt," or "red trousers," or "blue shirt," or "can," or "aluminum can." Furthermore, a segmented image should be understood as being able to depict two or more items. Therefore, labeling a segmented image may involve identifying two or more items for any given image. Identifying a drawn item can be done in various ways. For example, identifying a drawn item may be done by foreground-background segmentation. Alternatively, identifying a drawn item may be done by object instance segmentation. Object instance segmentation may be done by a machine learning model. For example, the machine learning model may be part of a classifier, or it may be the same machine learning model as the classifier. Therefore, object instance segmentation may be performed by a neural network such as a convolutional neural network or a fully connected neural network. After the images have been processed by a machine learning model that uses object instance segmentation, they may be provided to the classifier as training material. Images in any set of images may be subject to further image processing steps. For example, images in a set of segmented images may be subject to image enhancement, cropping, etc. Furthermore, it should be understood that captured images may be discarded from a set of images, for example, if the object is not identifiable at all or if the image quality is too low.

[0089] Furthermore, the processing unit 6 may be configured to perform a function of rejecting outlier images, i.e., images that deviate from the rest of the image set by a predetermined tolerance level. The rejection of outlier images may optionally be supplemented by manual intervention by the operator.

[0090] The lower half of Figure 1 shows a second sorting configuration 10, which sorts the input flow of items 12. The second sorting configuration 10 is spatially separated from the first sorting configuration 1. This should be understood as meaning that the first and second sorting configurations 1 and 10 sort different item flows 2 and 12. The first sorting configuration 1 may be located in the first sorting plant, while the second sorting configuration 10 may be located in the second sorting plant. Alternatively, the first and second sorting configurations 1 and 10 may be located in the same sorting plant. Therefore, this method can be implemented for sorting configurations located in different locations in plants, different locations in cities, different locations in countries, different countries, and / or different continents.

[0091] The second sorting configuration 10 is communicatively connected to the control unit 16. The control unit is communicatively connected to the processing unit 6 of the first sorting configuration 1. The control unit 16 may be connected to the processing unit 6 in various ways, such as via a network infrastructure by communicating through a cloud-based server. Therefore, classifiers may be distributed from the processing unit 6 to the control unit 16.

[0092] The control unit 16 can implement a machine learning model for the classifier to control the sorting process of the second sorting arrangement 10. Here, the input flow of item 12 is sorted into first and second sorted categories 13 and optionally into an unsorted category 14. Therefore, the input flow of item 12 in the second sorting arrangement 10 is sorted into the same number of sorted categories as the input flow of item 2 in the first sorting arrangement 1. Alternatively, the flows of items 2 and 12 may be sorted into different numbers of sorted categories 3 and 13. That is, the flow of item 2 in the first sorting arrangement 1 may be sorted into two sorted categories 3, while the flow of item 12 in the second sorting arrangement 10 may be sorted into one sorted category 13.

[0093] Figure 2 shows a flowchart of an exemplary method according to a first aspect of the concept of the present invention. In a first sorting setup 1, S1 is performed to sort the flow of items 2 into at least one sorted section 3 and optionally at least one unsorted section 4. Subsequently, S2 is performed to capture a set of section images of the items in at least one sorted section 3, and then S3 is performed to label the section images captured in the first sorting setup 1 with section labels. Each section label indicates the sorted section 3 of the at least one sorted section into which the depicted item was sorted. Labeling the section images S3 may further include identifying at least one depicted item for each section image, and the section label further indicates the identified item. Identifying at least one depicted item may be performed, for example, by object instance segmentation.

[0094] Subsequently, S4 is performed to build at least one classifier configured to associate items with categories. The at least one classifier is a machine learning model, and building at least one classifier S4 includes training the machine learning model using the category images and category labels taken in the first sorting arrangement 1. Building at least one classifier S4 may include building a separate classifier for each of the at least one sorted category 3.

[0095] As shown in Figure 2, the method may optionally include taking a set of stream images of items in the stream of items 2 in the first sorting setup 1 before sorting the stream of items 2 into at least one sorted category 3 and optionally at least one unsorted category 4. Furthermore, after taking the set of stream images S31, the set of stream images may be compared with the set of sorted images S32. Subsequently, the images depicting items depicted in any one of the sorted images are removed from the set of stream images S33, thereby generating a modified set of stream images. Then, each stream image of the modified set of stream images is labeled with a label indicating that the depicted item has been sorted into at least one unsorted category 4 S34. Thus, constructing at least one classifier S5 may further include training a machine learning model using the stream images of the modified set of stream images taken in the first sorting setup 1 and their respective associated labels.

[0096] This method further includes S5 capturing images of items in the flow of items 12 in a second sorting setup 10, and S6 controlling the sorting of items in the flow of items 12 to at least one sorted category 13 and optionally at least one unsorted category 14 using the images captured in the second sorting setup 10 and at least one classifier.

[0097] The method may further include S7 distributing at least one classifier from the first sorting configuration 1 to the second sorting configuration 10 via a network infrastructure such as a cloud-based server.

[0098] The method may further include updating at least one classifier distributed to the second sorting configuration 10 S8. Updating at least one classifier distributed to the second sorting configuration 18 includes repeating the steps of sorting S1, imaging S2, labeling S3, and construction S4 in the first sorting configuration 1, thereby providing at least one updated classifier. Furthermore, updating S8 includes distributing at least one updated classifier to the second sorting configuration 10 and activating at least one updated classifier in the second sorting configuration 10.

[0099] Updating at least one classifier distributed to the second sorting configuration 10 S8 may further include sending a verification request from the second sorting configuration 10 to the first sorting configuration 1, and if the verification request fails, the steps of sorting S1, imaging S2, labeling S3, and construction S4 are performed.

[0100] Updating at least one classifier S8 may further include training the machine learning model by intentionally making the machine learning model forget the data. This can be implemented in various ways. For example, training the machine learning model may further include embedding the data with the segmental labels associated with the segmental images taken in the first sorting arrangement, and optionally the data with the stream labels associated with the stream images taken in the first sorting arrangement, as data points in a spatial map such as a two-dimensional or three-dimensional map, and dividing the spatial map into cells, each cell having a data point density and a last input date and time indicating the date and time when the most recent data point was added to the cell. The data point density of a cell may also be called the embedding density. An exemplary two-dimensional spatial embedding map 20 is shown in Figures 3A-B.

[0101] Embedding data from an image set can be useful for a variety of reasons. For example, repeating the process of building at least one classifier may further include, for each cell, checking the data point density and / or last input date and time, and if the data density of the cell is greater than or equal to a predetermined threshold, and / or if the cell's last input date and time is the same as or older than the predetermined threshold, modifying the set of data points contained within the cell, for example, deleting all data points from the set of data points. Additionally or alternatively, repeating the process of building at least one classifier may further include, when adding a new data point to the spatial map, checking the data point density and / or last input date and time of the cell to which the data point is added, and if the cell's data density is greater than or equal to a predetermined threshold, discarding the new data point, and / or if the cell's last input date and time is the same as or older than the predetermined threshold, adding the new data point and deleting at least one neighboring data point within the cell.

[0102] This means that as the training dataset expands, redundant or outdated data points may be discarded. Therefore, at least one classifier may be updated to have control over the selection of new items, or to have control over the selection of items in a new way.

[0103] The above method should be understood as being implementable in two or more further sorting configurations. For example, in a third sorting configuration, the steps of taking images of items in the item flow and using the images taken in the third sorting configuration and at least one classifier to control the sorting of items in the item flow into at least one sorted category and optionally at least one unsorted category may be performed.

[0104] Instructions for implementing the exemplary method described above, and any variations thereof, may be stored in a computer-readable storage medium. Thus, the instructions can be used by the processing unit 6 to execute the method.

[0105] Figure 3A shows how data is spatially embedded in a two-dimensional spatial map 20 as data points 21. The data may be derived, for example, from a set of segmented images or stream images, or a set of segmented item images or streamed item images. That is, data points 21 may represent items depicted in segmented images and / or stream images. That is, data points 21 may additionally or alternatively represent items depicted in item images such as segmented item images or streamed item images. Thus, the placement of data points 21 in the map 20 can be a function of the class to which the items belong. For this purpose, data points may be placed in clusters 22, where each cluster 22 represents a class of items. In Figure 3A, data points 21 belonging to the same cluster 22 are shown in the same grayscale.

[0106] In Figure 3B, a grid is applied to map 20. The grid divides map 20 into cells 23. Each cell 23 has a data point density. Furthermore, each cell 23 may also have an associated last input date and time, which indicates the date and time the most recent data point was added to cell 23. Therefore, repeating the process of building at least one classifier may further include, for each cell 23, checking the data point density and / or last input date and time, modifying the set of data points 21 if the data density of cell 23 is above a predetermined threshold and / or the last input date and time of cell 23 is the same as or older than the predetermined threshold, for example, deleting all data points 21 from the set of data points 21 contained in cell 23. In other words, if the data point density of cell 23 is too high or the last input date and time is too old, the set of data points 21 contained in the cell may be modified accordingly. For example, if the last input date and time is too old, i.e., too much time has passed since the data points 21 were added to cell 23, all data points in cell 23 may be deleted. This can be useful, for example, when data points within a cell represent items that are no longer eligible for selection.

[0107] Additionally or alternatively, repeating the process of building at least one classifier may further include, when adding a new data point 21 to the spatial map 20, checking the data point density and / or last input date and time of the cell 23 to which the data point 21 is added, discarding the new data point 21 if the data density of cell 23 is greater than or equal to a predetermined threshold, and / or adding the new data point 21 and removing at least one neighboring data point 21 in cell 23 if the last input date and time of cell 23 is the same as or older than a predetermined threshold.

[0108] The predetermined threshold for the final input date and time may be, for example, one week, one month, multiple months, one year, or multiple years.

[0109] A predetermined threshold for data density may be, for example, a number assigned to cell 23, each number such as 10, 20, or 100, indicating that cell 23 can only contain 10, 20, or 100 data points. All cells 23 may have the same threshold. Cell 23 may have a different threshold than other cells 23.

[0110] Alternatively, a predetermined threshold for cell 23 may be set according to a specified ratio, which is calculated by dividing the total number of data points 21 in all cells 23 by the number of cells 23. That is, a predetermined threshold for a cell may be 10 data points 21 per cell, 20 data points 21 per cell 23, or 100 data points 21 per cell 23. Therefore, the exact value may vary depending on how many data points 21 the dataset contained in total at the time the predetermined threshold was set. Furthermore, the predetermined threshold may change over time. The predetermined threshold may be refreshed, for example, each time at least one classifier is updated, so that each cell 23 is provided with an updated ratio.

[0111] Alternatively, the predetermined threshold may be set according to the probability distribution of data points 21 on the spatial map 20. That is, cells 23 to which data points 21 are frequently added, or cells 23 that already contain many data points 21, may have a higher predetermined threshold than cells 23 to which data points 21 are not added as frequently, or cells with few data points. Thus, if a new data point 21 is provided to a cell 23 that has been assigned a low probability density value, the predetermined thresholds of all other cells 23, or at least all other cells 23 with a higher probability density value, may be increased.

[0112] Additionally or alternatively, particularly when updating at least one classifier, the dataset may be cleaned up before any further data points 21 are added. For example, the dataset may be cleaned up at a certain percentage per cell 23. For instance, 5%, 10%, 20%, or 30% of all data points 21 in each cell 23 may be removed before any further data points 21 are provided to any cell.

[0113] Figure 4 shows a schematic perspective view of an exemplary second sorting configuration 700 in which at least one classifier is implemented, i.e., at least one classifier is distributed to this second sorting configuration 700.

[0114] A stream of items 710, such as textile materials like clothing, is supplied to the sorting arrangement 700. The material 710 is transported through the detection area 720 by a conveying device such as a conveyor belt or slide. However, the material may be provided through the detection area 720 by any suitable means or manually without any technical means. A light source configuration 730 and an inspection system 740, including a camera system 750 and optionally a spectrometer system 760, are provided inside the housing 222. The camera system 750 and optionally the spectrometer system 760 are adapted to receive and analyze light, which is emitted by the light source configuration 730 and subsequently reflected and / or scattered by the material in the detection area 720. The light source configuration 730 typically emits a spectrum within the wavelength range of ultraviolet, visible light and / or near-infrared light. The spectrometer system 760 acquires spectra typically within the ultraviolet, visible, and / or near-infrared wavelength ranges, and the camera system 750 acquires images typically within the ultraviolet, visible, and / or near-infrared wavelength ranges. The inspection system 740 of the sorting arrangement 700 is configured to distinguish a group of fibrous materials from other materials based on the acquired spectra and / or acquired images. In other words, the system 740 may be configured to distinguish a particular type of material or group of materials from other types of materials based on its spectrum, including its color, and / or its shape, size, or other detectable appearance.

[0115] The sorting arrangement 700 may optionally include a laser triangulation system 746 configured to determine height information related to the material being transported through the detection area 720.

[0116] Furthermore, the sorting configuration 700 preferably includes an discharge system 224 (such as a robotic arm or a nozzle for spraying pressurized air) for sorting materials into different categories based on images captured by the camera system and at least one classifier. That is, at least one classifier may be implemented in a control unit 16 located in the sorting configuration 700, and the control unit 16 is also configured to perform functions for physically sorting items, such as operating a robotic arm or nozzle. Thus, such a control unit 16 may be communicatively connected to an inspection system including a light source configuration 730, as well as the camera system 750 and optionally a spectrometer system 760, and may additionally be configured to operate such a system. Additionally or alternatively, information received from the inspection system and optionally a triangulation system may be used to verify the sorting decision made based on the classifier or to enable sorting into finer categories. Thus, images captured in the second sorting configuration 700 and at least one classifier are used when sorting the flow of items into at least one sorted category. A more detailed description of the sorting arrangement 700, including the light source configuration, inspection system, laser triangulation system, and discharge system 224, can be found in International Publication No. 2015 / 063300, which is incorporated herein by reference.

[0117] The drawings and specification disclose preferred modifications and embodiments of the concept of the present invention, and certain terms are used, but they are used only in a general and descriptive sense and are not intended to be limiting, and the scope of the concept of the present invention is set out in the following claims.

Claims

1. A method for controlling the sorting of items in multiple sorting arrangements, wherein the method is: In the first sorting arrangement (1), The item flow (2) is sorted into at least one sorted category (3) and optionally at least one unsorted category (4) (S1), (S2) taking a set of classification images of the items within the at least one selected classification (3), The division images captured in the first sorting arrangement (1) are labeled with division labels (S3), and each division label indicates the selected division (3) among the at least one selected division (3) from which the depicted item was sorted. (S4) Constructing at least one classifier configured to associate items with categories, wherein the at least one classifier is a machine learning model, and constructing the at least one classifier includes training the machine learning model using the category images and category labels captured in the first sorting arrangement. A method comprising: capturing images of items in an item flow (12) in a second sorting arrangement (10) (S5); and controlling the sorting of items in the item flow (12) into at least one sorted category (13) and optionally at least one unsorted category (14) using the images captured in the second sorting arrangement (10) and the at least one classifier (S6).

2. The method according to claim 1, wherein labeling (S3) includes identifying at least one depicted item for each segment image by object instance segmentation or the like, and each segment label further indicates each identified item.

3. The process further includes extracting a set of category item images from the aforementioned set of category images, wherein each category item image depicts one identified item such that the number of category item images in the set of category item images is the same as the number of items identified within the set of category images. The method according to claim 2, wherein labeling (S3) is performed on the set of classification item images, and training the machine learning model is performed using the classification item images and classification labels captured in the first sorting arrangement.

4. In the first sorting arrangement described above, before sorting the item flow (2) into at least one sorted section (3) and optionally at least one unsorted section (4), the process further includes taking a set of stream images of the items in the item flow (2) (S31) and labeling the stream images with stream labels (S32), The method according to claim 2 or 3, wherein labeling (S32) includes identifying at least one drawn item for each stream image, and the stream label further indicates each identified item.

5. Labeling (S32) further includes extracting a set of stream item images from the set of stream images, wherein each stream item image depicts one identified item such that the number of stream item images in the set of stream item images is the same as the number of items identified in the set of stream images. The method according to claim 4, wherein labeling (S32) is performed on the set of stream item images.

6. The method involves comparing the set of stream images with the set of segmented images, or comparing the set of stream item images with the set of segmented item images, deleting an image that depicts an item depicted in either the segmented image or the segmented item image from the set of stream images or the set of stream item images, thereby generating a modified set of stream images or a modified set of stream item images (S33). The process further includes labeling each stream image in the modified set of stream images, or each stream item image in the modified set of stream item images, with a label indicating that the depicted item has been sorted into at least one unsorted category (4) (S34), The method of claim 4 or 5, wherein constructing the at least one classifier (S5) further comprises training the machine learning model using the stream images of the modified set of stream images or the stream item images of the modified set of stream item images captured in the first sorting arrangement (1) and the labels associated with each.

7. The method according to any one of claims 1 to 6, wherein labeling the segmented images captured in the first sorting arrangement (1) with segmented labels (S3) is performed by a processing unit (6), and the processing unit (6) is further configured to provide the segmented images captured in the first sorting arrangement and the segmented labels to the machine learning model.

8. The method according to claims 6 and 7, wherein the step described in claim 6 is performed by the processing unit (6), and the processing unit is further configured to provide the machine learning model with the stream images of the modified set of stream images or the stream item images of the modified set of stream item images captured in the first sorting arrangement (1), and the labels associated with each of them.

9. The method according to any one of claims 1 to 6, further comprising distributing the at least one classifier from the first sorting arrangement (1) to the second sorting arrangement (10) via a network infrastructure such as a cloud-based server (S7).

10. The method further includes updating the at least one classifier distributed to the second sorting arrangement (10) (S8), which means sorting the flow of items in the first sorting arrangement (1) into at least one sorted category (3) and optionally at least one unsorted category (4) (S1). To take a set of classification images of the items within the at least one selected classification (3) (S2), Labeling the segmented images captured in the first sorting arrangement (1) with segmented labels (S3), and The process of constructing at least one classifier (S4) by training the machine learning model using the segmented images and segmented labels captured in the first sorting arrangement (1) is repeated, thereby providing at least one updated classifier. Distributing the at least one updated classifier to the second sorting arrangement (10), The method according to claim 9, further comprising activating the at least one updated classifier in the second sorting configuration (10).

11. The method according to claim 10, wherein updating the at least one classifier distributed to the second sorting configuration (10) (S8) further includes sending a verification request from the second sorting configuration (10) to the first sorting configuration (1), and if the verification request fails, the process according to claim 10 is performed.

12. Training the aforementioned machine learning model is The data of the segmented image captured in the first sorting arrangement (1) and the associated segmented label, and optionally the data of the stream image captured in the first sorting arrangement (1) and the associated stream label, are embedded as data points in a spatial map such as a two-dimensional or three-dimensional map. The method according to any one of claims 1 to 11, comprising dividing the spatial map into cells, each cell having a data point density and a last input date and time indicating the date and time when the latest data point was added to the cell.

13. Repeating the above step of constructing at least one classifier (S4) is, The method according to claims 10 and 12, wherein for each cell, the data point density and / or the last input date and time are checked, and if the data density of the cell is equal to or greater than a predetermined threshold, and / or the last input date and time of the cell is the same as or older than the predetermined threshold, the set of data points contained in the cell is modified, for example, by deleting all data points from the set of data points.

14. Repeating the above step of constructing at least one classifier (S4) is, When adding a new data point to the spatial map, the data point density and / or the last input date and time of the cell to which the data point is added are confirmed. The method according to claims 10 and 12, or claim 13, further comprising discarding the new data point if the data density of the cell is greater than or equal to a predetermined threshold, and / or adding the new data point and deleting at least one neighboring data point in the cell if the last input date and time of the cell is the same as or older than a predetermined threshold.

15. The method according to any one of claims 1 to 14, wherein constructing at least one classifier (S4) includes constructing a classifier for each of the at least one sorted section (3) sorted section (3).

16. A computer-readable storage medium storing instructions for implementing the method according to any one of claims 1 to 14.

Images